Building a Production RAG Pipeline That Actually Cites Its Sources

A RAG demo takes an afternoon. A RAG system people actually rely on takes a lot more care — because the failure mode isn't a crash, it's a confident, wrong answer. The goal is grounding: every claim should trace back to a source you can point at.

Chunking is a product decision, not a default

The instinct is to split documents on a fixed token count and move on. But chunk boundaries decide what the model can and can't see together. I lean toward structure-aware splitting — headings, sections, tables stay intact — with a small overlap so context doesn't get amputated mid-thought.

• Preserve semantic units (sections, list items) over fixed windows.
• Keep a little overlap, but not so much that you bloat the index.
• Store rich metadata: source, section, page — you'll need it for citations.

Hybrid search beats pure vectors

Dense vectors are great at meaning and bad at exact terms — names, error codes, SKUs. Keyword search is the opposite. Combining both (and reranking the merged set) consistently retrieves the right context more often than either alone.

If the model can't find it, the model can't say it. Retrieval quality caps answer quality.

Force the grounding

I prompt the model to answer only from retrieved context and to attach citations inline. When confidence is low or context is thin, the right answer is "I don't know" — and that's a feature. A system that admits uncertainty earns the trust to be used at all.

System: Answer ONLY from the provided context.
Cite each claim as [source:section]. If the context is
insufficient, say so plainly. Do not invent details.

None of this is exotic. It's discipline: good chunks, hybrid retrieval, strict grounding, and a willingness to say "not enough information." That's the difference between a clever demo and something a team can depend on.