Databricks, GenAI & What I’m Learning

My journey into GenAI engineering made me realise something fairly quickly:

An AI application is only as good as the data and infrastructure behind it.

That led me deeper into Databricks, RAG, vector search and data pipelines—areas that were initially outside my traditional UX comfort zone.

From the interface to the data

As a UX designer, I naturally start with the user:

What does the person need?
What problem are we solving?
What should the experience feel like?

Working with GenAI added another layer:

Where does the AI get its knowledge?

That question led me to explore the data pipeline behind a RAG application.

I started working with the Medallion architecture:

Landing → Bronze → Silver → Gold

The concept was surprisingly familiar.

It reminded me of good UX practice:

Raw information → structured information → useful information → actionable experience

The difference is that this time I was building the infrastructure that makes that transformation possible.

Learning RAG

I experimented with a pipeline that takes structured data, creates chunks, generates embeddings and stores them for retrieval.

The flow became:

Data → Chunking → Embeddings → Vector Search → Retrieval → LLM → Response

I worked with Databricks Vector Search to understand how semantic retrieval works and how an LLM can be grounded in organisational data rather than relying only on its pretrained knowledge.

This changed how I think about AI interfaces.

When a user asks a question, the quality of the answer isn't determined only by the model.

It depends on:

• What data was ingested

• How it was structured

• How it was chunked

• How embeddings were generated

• What the vector search retrieved

• How the context was passed to the model

• How the response was evaluated

The UX begins long before the user sees the answer.

The uncomfortable part: debugging

Learning Databricks has also been a reminder that GenAI engineering isn't always glamorous.

I've encountered:

• Vector Search configuration issues

• Index creation problems

• Schema mismatches

• Endpoint limitations

• Library and dependency issues

• Retrieval problems

• Infrastructure constraints

And honestly, this has been valuable.

Each problem forced me to understand what was happening underneath the abstraction.

Instead of simply saying "the RAG isn't working," I started asking:

Is the data correct?
Is the schema correct?
Are the chunks meaningful?
Are the embeddings appropriate?
Is retrieval returning the right context?
Is the index synchronised?

That's engineering thinking.

What Databricks taught me about UX

The biggest lesson isn't about Databricks itself.

It's about systems thinking.

A user experiences one simple interaction:

"Show me the relevant information."

Behind that seemingly simple request might be:

Data ingestion → Transformation → Storage → Embedding → Indexing → Retrieval → LLM → Evaluation → UI

Good AI UX needs to hide this complexity without ignoring it.

Where I'm going next

I'm continuing to connect the pieces:

UX & Service Design
↓
GenAI Product Design
↓
RAG & Data Engineering
↓
Agentic Systems
↓
Evaluation & Observability
↓
AI Governance

I'm not trying to become a traditional data engineer.

I'm learning enough of the AI/data stack to understand, design and build intelligent products responsibly.

My biggest learning

AI is not just an intelligence problem. It's a data, systems, product and experience problem.

Databricks has helped me see the part of the AI product that users never see—but that ultimately determines what they experience.