Query decontextualization for multi-turn chat

TL;DR: When users ask follow-up questions like "Does it include WhatsApp?", AskVault rewrites them to "Does AskVault pricing include WhatsApp?" before vector search runs. Retrieval precision on multi-turn conversations climbs 15-30% with one extra ~120ms LLM call. Same technique Cloudflare AutoRAG calls "automatic query decontextualization." Enabled by default on every workspace.

The problem

A user opens your chat widget and types two messages:

Turn 1: "What's your pricing?"
Bot:    "Starter is $29/mo, Growth $99/mo..."
Turn 2: "Does it include WhatsApp?"

The vector embedding of the second message has no signal about what "it" refers to. Searching the vector index for "Does it include WhatsApp?" retrieves chunks about WhatsApp generally — Twilio setup, message templates, API docs — not chunks about your pricing plans.

Result: the LLM answers correctly only by luck (because the conversation history is in its context), or it cites the wrong page, or it says "I don't see WhatsApp in our pricing" because the relevant pricing chunk never made it into the top-K.

This is the single largest accuracy bug in naive RAG pipelines.

How AskVault fixes it

Before the vector search runs, AskVault rewrites the user's message into a standalone query using the last few turns as context:

Original:    "does it include WhatsApp?"
Decontextualized: "Does AskVault's Starter plan pricing include WhatsApp?"

The rewritten query is what hits the vector index. The pricing chunks come back. The agent answers correctly with the right source citation.

When the rewrite happens (and when it skips)

Condition	Rewriter behavior
First turn of a conversation	Skipped — message is already standalone
Message is a greeting/thanks ("hi", "ok", "thanks")	Skipped via heuristic
Message is one or two words	Skipped — no referential signal to resolve
Multi-turn follow-up with prior history	Rewritten via one Gemini-flash-lite call

You can flip the whole feature off with QUERY_REWRITER_ENABLED=false in your environment if you ever need to fall back.

What model runs the rewrite

A cheap fast model (gemini-2.5-flash-lite by default), not your workspace's configured premium chat model. This means:

• The rewrite call is not metered against your customer's monthly chat quota.
• Median latency is ~120 milliseconds added to the turn — well inside the perception threshold.
• On any failure (timeout, parse error, network), the original query is passed through unchanged. The chat turn never fails because of decontextualization.

What you'll see in your data

If you watch a workspace with active multi-turn conversations:

• Retrieval precision on follow-up turns rises 15-30% (published RAG benchmark numbers; varies by knowledge base shape).
• Source citation accuracy improves — the agent attributes answers to the page that actually contained the answer instead of an adjacent page that happened to be lexically close.
• TTFT increases ~120ms on follow-up turns only. First turns are unaffected (no history to fold).

Configuration

Three settings in backend/app/config.py:

QUERY_REWRITER_ENABLED: bool = True          # global kill switch
QUERY_REWRITER_MODEL: str = "gemini-2.5-flash-lite"
QUERY_REWRITER_MAX_TURNS: int = 4            # how many prior turns to include

QUERY_REWRITER_MAX_TURNS defaults to 4 because beyond that the rewriter tends to over-fold context (compressing too much). Bump to 6 for long sales-cycle conversations; drop to 2 for single-question support flows.

Why this matters more in 2026

As chat surfaces shift from one-shot Q&A to genuine multi-turn dialogue (the trend in support tickets, lead qualification, agentic workflows), the percentage of queries that are follow-ups climbs sharply. A pipeline that's 90% accurate on first-turn queries but 60% accurate on follow-ups feels broken to users.

Query decontextualization is one of those low-cost, high-leverage RAG fixes that every serious chat system runs. AskVault has it on by default; competitors like Chatbase and SiteGPT pass the raw query to vector search and rely on the LLM to recover, which works some of the time.

Implementation reference

Source: backend/app/services/query_rewriter.py. Wired into services/channel_router.py at both call sites (streaming and non-streaming RAG paths). 19 unit tests in tests/test_query_rewriter.py cover the parser, the heuristic skip list, the artefact stripping, and the graceful-degradation path on LLM failure.