RAG is not a product: retrieval patterns that hold up
Chunk, embed, search, stuff into a prompt. That pipeline gets you a convincing prototype and a support queue full of confidently wrong answers. Here is what we build instead.
Retrieval-augmented generation is a technique, not an architecture. The default recipe — split documents into 500-token chunks, embed them, take the top five by cosine similarity — is the weakest version of it, and it is what most teams ship.
Chunking is a modelling decision
Fixed-size chunks cut tables in half and separate headings from the text they govern. Chunk along the document's own structure instead: sections, clauses, list items. Keep the heading trail in the chunk so a retrieved fragment still says what it belongs to.
Hybrid search beats embeddings alone
Vector search is bad at exact identifiers — part numbers, error codes, policy names — precisely the terms users type. Run keyword search alongside it and fuse the rankings. The extra query costs a few milliseconds and removes a whole category of misses.
Rerank before you generate
Retrieve twenty candidates, rerank them with a cross-encoder, pass the best four to the model. Retrieval recall and prompt precision are different problems and deserve different stages.
Answer with citations or refuse
Require the model to ground every claim in a retrieved passage, and to say plainly when the corpus does not cover the question. A system that says "I don't have that" is worth far more than one that improvises.
- Structure-aware chunking with heading context
- Hybrid keyword + vector retrieval, fused
- Cross-encoder reranking before generation
- Mandatory citations and an explicit refusal path
- A golden question set you re-run on every corpus change
The last one matters most. Corpora drift, documents get replaced, and yesterday's good answer quietly becomes today's wrong one.
