RAG beyond the demo: retrieval quality is the product — AI Robos
HOME / BLOG / ARTICLE
AI ENGINEERINGMAY 7, 2026 · 11 MIN READ

RAG beyond the demo: retrieval quality is the product.

When a RAG system disappoints, teams reach for a bigger model. In our experience the model is almost never the problem — the retrieval layer is. Here's how we build and measure it.

A language model can only be as good as what you hand it. If the right passage isn't in the context window, no amount of model quality will produce a grounded answer — it will produce a confident one. That single sentence explains most disappointing RAG deployments we're called in to fix.

1. Chunk along the document's own structure

Fixed 512-token windows are the default everywhere and wrong almost everywhere. Contracts break at clauses, policies at numbered sections, articles at paragraphs with their headline context attached. On a 1.2M-article publisher archive, re-chunking along editorial structure — headline + dek + paragraph, with section metadata — improved answer usefulness more than any model upgrade we tested. Chunking is unglamorous, cheap to redo, and the highest-leverage dial in the stack.

2. Hybrid search, then rerank

Vectors miss exact identifiers — part numbers, statute references, people's names. Keyword search misses paraphrase. Production systems need both, fused, and then a reranker ordering the merged candidates by actual relevance to the question. The pattern is boring and it wins: broad hybrid recall, aggressive rerank, small final context. Stuffing forty mediocre chunks into a long context window is how you pay more per query for worse answers.

Recall gets the right passage into the room. Reranking gets it a seat at the table. The model just writes the minutes.

3. Measure retrieval separately from generation

Most teams evaluate end-to-end answers only, so they can't tell whether a failure is retrieval or generation. We maintain two suites:

Retrieval evals: golden questions with known source passages — did the right chunks appear in the top k? Track recall@k per query class.
Answer evals: given correct retrieval, is the answer faithful, complete, and properly cited?

The split diagnoses in minutes what end-to-end scores obscure for weeks. Retrieval regressions — from a re-index, an embedding change, a new document format — get caught in CI, not by users.

4. No citation, no claim

The last layer is a policy, not a technique: the system may not state what it cannot cite. Enforcing that — with refusal as an acceptable output and entitlement filters applied at query time, not after generation — is what turns a clever search box into something a newsroom, a law firm, or a compliance team will actually rely on. Users forgive "I couldn't find that." They don't forgive being lied to with confidence.

Ashish
Founder & CEO, AI Robos
Talk to us
NEXT ARTICLE
The boring parts that make AI agents work in production →
All articles