A language model can only be as good as what you hand it. If the right passage isn't in the context window, no amount of model quality will produce a grounded answer — it will produce a confident one. That single sentence explains most disappointing RAG deployments we're called in to fix.
1. Chunk along the document's own structure
Fixed 512-token windows are the default everywhere and wrong almost everywhere. Contracts break at clauses, policies at numbered sections, articles at paragraphs with their headline context attached. On a 1.2M-article publisher archive, re-chunking along editorial structure — headline + dek + paragraph, with section metadata — improved answer usefulness more than any model upgrade we tested. Chunking is unglamorous, cheap to redo, and the highest-leverage dial in the stack.
2. Hybrid search, then rerank
Vectors miss exact identifiers — part numbers, statute references, people's names. Keyword search misses paraphrase. Production systems need both, fused, and then a reranker ordering the merged candidates by actual relevance to the question. The pattern is boring and it wins: broad hybrid recall, aggressive rerank, small final context. Stuffing forty mediocre chunks into a long context window is how you pay more per query for worse answers.
Recall gets the right passage into the room. Reranking gets it a seat at the table. The model just writes the minutes.
3. Measure retrieval separately from generation
Most teams evaluate end-to-end answers only, so they can't tell whether a failure is retrieval or generation. We maintain two suites:
The split diagnoses in minutes what end-to-end scores obscure for weeks. Retrieval regressions — from a re-index, an embedding change, a new document format — get caught in CI, not by users.
4. No citation, no claim
The last layer is a policy, not a technique: the system may not state what it cannot cite. Enforcing that — with refusal as an acceptable output and entitlement filters applied at query time, not after generation — is what turns a clever search box into something a newsroom, a law firm, or a compliance team will actually rely on. Users forgive "I couldn't find that." They don't forgive being lied to with confidence.