Conversational RAG: Building a Production-Ready RAG Pipeline

Introduction

Retrieval-Augmented Generation (RAG) has quickly become the standard architecture for building enterprise AI assistants. Whether you’re developing an HR chatbot, an internal documentation assistant, or a customer support agent, the overall pipeline usually looks familiar:

1. Convert the user’s question into an embedding.
2. Search a vector database for relevant documents.
3. Send the retrieved context to a Large Language Model (LLM).
4. Generate an answer grounded in the retrieved information.

For straightforward, standalone questions, this approach works remarkably well.

A user asks:

“What is our annual leave policy?”

The system retrieves the relevant policy document, sends it to the LLM, and returns an accurate response. The problems begin once the conversation becomes conversational. Real users rarely ask isolated questions. Instead, they ask follow-up questions, refine their intent, compare policies, or simply acknowledge the response before asking the next question. A typical interaction looks more like this:

User:
What is the annual leave policy in the UAE?

Assistant:
...

User:
How about Saudi Arabia?

Assistant:
...

User:
What about probation employees?

Assistant:
...

User:
Thanks!

To a human, every question is perfectly clear because the previous conversation provides the missing context. To a traditional RAG pipeline, however, each message is treated as a completely independent search query. That seemingly small assumption creates one of the biggest weaknesses in production RAG systems.

The Hidden Problem with Traditional RAG

Most first-generation RAG implementations are designed around single-turn retrieval. Every incoming message follows roughly the same pipeline:

Although the LLM may receive the full conversation history, the retrieval layer does not.

Instead, it embeds only the latest user message before searching the vector database.

This creates a disconnect:

• The generator is multi-turn.
• The retriever is single-turn.

That architectural mismatch is subtle, but it has a major impact on answer quality.

Why Follow-up Questions Break Retrieval

Imagine an employee interacting with an internal policy assistant.

Turn 1

What is the annual leave policy in the UAE?

Retrieval works exactly as intended.

The embedding contains rich semantic information:

• annual leave
• policy
• UAE

The vector database easily returns the correct document.

Turn 2

How about Saudi Arabia?

From a user’s perspective, the meaning is obvious.

They’re asking:

“What is the annual leave policy in Saudi Arabia?”

Unfortunately, that isn’t what the retriever sees.

The embedding is generated from only three words:

How
about
Saudi Arabia

The concepts annual leave and policy have disappeared entirely.

Instead of searching for the correct HR policy, the retriever may return:

• relocation policy
• travel policy
• payroll documents
• random regional documents
• or nothing relevant at all

The LLM now receives poor context.

And regardless of how capable the model is, it can only reason over the information it has been given.

Garbage retrieval leads to poorly grounded answers.

The same problem appears with even shorter messages.

Is it 30 days?

Does this apply to contractors?

What about interns?

Can I carry it forward?

Thanks!

None of these are meaningful search queries in isolation.

Yet a traditional RAG system still performs:

• embedding generation
• vector search
• reranking
• document injection

for every single message. The result is:

• irrelevant retrieved chunks
• unnecessary API calls
• increased latency
• higher inference cost
• confusing citations
• poorer user experience

Ironically, many engineering teams respond by tuning embeddings, changing chunk sizes, or switching vector databases, even though the real issue lies before retrieval ever begins.

The Realization: Retrieval Queries Are Not Conversation

One of the biggest lessons from redesigning our RAG pipeline was this:

A user’s message and a retrieval query are not always the same thing.

Humans naturally communicate using context. Search engines do not. These two goals are fundamentally different:

User message	Retrieval query
Natural conversation	Explicit search intent
May rely on previous context	Must be self-contained
Optimized for readability	Optimized for retrieval
Written by humans	Generated for the vector database

This distinction changes how you think about conversational AI.

Instead of embedding exactly what the user typed, the system should first ask:

“What is the user actually trying to retrieve?”

That realization became the foundation for redesigning the entire retrieval pipeline.

Rethinking the Architecture

Rather than treating every incoming message identically, we introduced a lightweight orchestration layer before retrieval. Instead of immediately searching the vector database, each message first goes through a decision-making process.

Instead of assuming every message requires document retrieval, the system first determines:

• Is this a greeting?
• Is this small talk?
• Is this a follow-up question?
• Does this message even require searching the knowledge base?

Only then does it decide whether retrieval is necessary and, if so, what the retrieval query should actually be. This layered approach introduces query routing, query condensation, and separate conversational handling before retrieval, addressing the limitations of the earlier single-turn pipeline.

This seemingly simple change dramatically improves both answer quality and efficiency.

The New Pipeline

Compared to a traditional RAG pipeline, only two new components have been introduced:

• Query Router
• Query Condensation

Everything else, embeddings, vector search, reranking, and LLM synthesis, remains largely unchanged. The biggest improvement comes not from replacing the retrieval stack, but from feeding it better inputs.

Layer 1 - Query Routing

The first decision the system makes is surprisingly simple:

Does this message even need retrieval?

Not every user message is a knowledge retrieval request.

Consider the following examples:

User Message	Retrieval Required?
Hello	No
Thanks!	No
Good morning	No
Can you help me?	No
What is the maternity leave policy?	Yes
Is it different in Germany?	Yes

How the Router Works

The router performs a lightweight classification step. Its job is not to answer the user’s question. Its job is simply to decide:

Is this message:

A. Conversational

or

B. Document Retrieval?

Typical conversational messages include:

• greetings
• thanks
• acknowledgements
• casual conversation
• clarification requests that don’t require documents

Everything else proceeds down the document retrieval path.

Avoiding an LLM Call When Possible

One interesting design choice is that not every routing decision needs an LLM. Many conversational messages are obvious.

Hi

Hello

Thanks

Goodbye

Good morning

See you later

These can be identified using simple heuristics or pattern matching.

This means:

• zero token usage
• zero model latency
• deterministic behaviour

Only ambiguous messages require an LLM classifier. For example:

Can you explain that?

Can you elaborate?

I'm confused.

That doesn't seem right.

These depend on context and may require an inexpensive classification model to determine whether retrieval is needed.

The original implementation follows this hybrid approach: obvious conversational messages are handled through fast heuristics, while borderline cases can optionally use a lightweight LLM classifier. When confidence is low, the system safely defaults to the document retrieval path rather than risking a missed search.

Designing for Safety

An important principle when building production systems is:

False positives are usually worse than false negatives.

Imagine the router mistakenly classifies this as conversational:

“What’s the bereavement leave policy?”

Instead of retrieving documentation, the assistant might respond from general knowledge. That’s a dangerous failure mode. A much safer strategy is:

If confidence is low -> Always choose document retrieval

The worst case becomes an unnecessary retrieval. The alternative could be an ungrounded answer.

Layer 2 - Query Condensation

If the router decides the message requires document retrieval, the next question becomes:

Should we search using the user’s exact words?

Often, the answer is no. Consider this conversation:

User: What is the annual leave policy in the UAE?

Assistant: ...

User: How about Saudi Arabia?

The user isn’t actually searching for:

How about Saudi Arabia?

They’re searching for:

What is the annual leave policy in Saudi Arabia?

Those are completely different retrieval queries.

What Is Query Condensation?

Query condensation rewrites short follow-up questions into fully self-contained search queries. It uses the conversation history to recover the missing context. Example:

User: What is annual leave in UAE?

↓

Latest Message: How about Saudi Arabia?

↓

Condensed Query: What is the annual leave policy in Saudi Arabia?

Notice something important. The user’s original message is never changed. Only the retrieval query changes.

The assistant still responds to:

“How about Saudi Arabia?”

The vector database simply receives a much better search query. This is exactly how the source implementation works: short follow-up questions are rewritten into standalone retrieval queries using the conversation history, while the original user message is preserved for both chat history and response generation.

When Should Condensation Run?

Not every message benefits from rewriting. For example:

Explain the parental leave policy in detail.

This is already a perfectly valid retrieval query.

Rewriting it adds:

• latency
• cost
• another potential failure point

without improving retrieval quality. A good strategy is to only run condensation for:

• short follow-up questions
• messages with missing context
• pronoun-heavy questions
• comparisons
• clarifications

Examples include:

How about Germany?

What about contractors?

Is that different?

Can they also apply?

Does it expire?

What if I'm part-time?

These messages rely on earlier turns to make sense.

Why This Works

Embeddings perform best when the semantic intent is explicit. Compare these two retrieval queries. Poor retrieval query:

How about Germany?

Improved retrieval query:

What is the annual leave policy in Germany?

The second query contains:

• policy
• annual leave
• Germany

Every important concept is now present in the embedding. The retriever no longer has to infer missing context.

Layer 3 - The Conversational Path

Once routing has been introduced, an interesting optimisation becomes possible. If a message is purely conversational:

Hello!

Thanks!

Good morning!

That was helpful.

Why perform retrieval at all? There is nothing to retrieve. Instead, these messages bypass the RAG pipeline entirely.

Message -> Router -> Conversational -> Direct LLM -> Response

No embedding. No vector search. No reranker. No injected document context. The assistant simply generates a natural conversational response. This dedicated conversational path eliminates unnecessary retrieval work and avoids irrelevant citations for greetings and acknowledgements.

Layer 4 - RAG Synthesis

Interestingly, this layer changes the least. The LLM still receives:

• retrieved chunks
• conversation history
• system instructions
• citation rules

The only difference is that retrieval quality has already improved. Instead of synthesising answers from weak or irrelevant context, the model now receives documents that better match the user’s actual intent. This illustrates an important principle of RAG architecture:

Better retrieval almost always leads to better generation.

Many hallucination problems are, in reality, retrieval problems.

Key Takeaways

At first glance, adding routing and query condensation might seem like a small architectural refinement. In practice, these two layers fundamentally change how a conversational RAG system behaves. Instead of treating every message as a search query, the system now understands that different messages serve different purposes:

• Some are conversations.
• Some are document searches.
• Some need additional context before retrieval.
• Some don’t need retrieval at all.

That distinction allows the rest of the RAG pipeline to operate on cleaner, more meaningful inputs, without replacing the existing retrieval infrastructure.

Coming Soon

This article covered how to transform a traditional single-turn RAG pipeline into a conversational RAG system using query routing and query condensation. In the next part, we’ll go beyond the query pipeline and explore what it takes to run conversational RAG in production.

We’ll cover document ingestion strategies, metadata enrichment, chunking best practices, evaluation methodologies, cost and latency analysis, and the trade-offs between architectural complexity and business value. You’ll also learn how to measure retrieval quality, build an evaluation dataset, and decide when a layered conversational RAG architecture is worth the investment.

The next part will include code examples for implementing query routing, query condensation, retrieval orchestration, and evaluation, along with practical engineering patterns you can apply to your own RAG applications.