Retrieval-augmented generation is the default architecture for putting an LLM in front of a company's own knowledge, and for good reason: it grounds answers in real documents, keeps data out of model weights and updates as fast as your content does. It's also the architecture behind most "the AI made something up" incidents we get called in to fix. When we dig in, the model is rarely the culprit. The retrieval step handed it the wrong context, or no context, and the model did what models do — filled the gap fluently.
Here's what separates RAG systems that earn trust from the ones that quietly lose it.
Start with the corpus, not the vector database
Enterprise knowledge is not clean markdown. It's PDFs with tables that shatter on extraction, slide decks where the meaning lives in the diagram, ten versions of the same policy with one still in force, and pages where the title is the only thing that identifies the product. Before choosing an embedding model, spend the time on ingestion: layout-aware parsing, table preservation, deduplication, version resolution and metadata (owner, date, product, region) attached to every chunk. A RAG system is only as good as the worst document it retrieves from with confidence.
Chunk for meaning, and keep the breadcrumbs
Fixed 500-token windows are easy and wrong. Chunks should follow the document's own structure — sections, clauses, table rows — and each one should carry the headings above it so "the fee is 2%" doesn't lose which plan it belongs to. Two techniques pay off repeatedly: store small chunks for precise matching but return the surrounding parent section to the model, and prepend a one-line generated summary of the document to each chunk so context travels with it.
Hybrid search is the baseline, not the upgrade
Pure vector search misses exact identifiers — part numbers, error codes, people's names, clause references — that keyword search nails, while keyword search misses paraphrases that embeddings catch. Production systems run both and fuse the results, then apply a reranker to the top candidates. Add metadata filtering (the right region, the current version, documents this user is allowed to see) before semantic ranking, not after. Access control at retrieval time isn't optional in an enterprise; the model must never see a document the user couldn't open.
Make the model show its work
Grounding is a contract between retrieval and generation: answer only from the supplied passages, cite which ones, and say "I don't have that" when they don't cover the question. Enforce it in the prompt, verify it after — a lightweight check that every claim in the answer is supported by a cited chunk catches most hallucinations before the user does. Citations aren't a UX nicety; they're how a user with a compliance question decides whether to trust the answer, and how you debug the ones they didn't.
Evaluate retrieval separately from generation
When an answer is wrong, you need to know which half failed. Build a golden set of real questions with the passages that should be retrieved, and track retrieval metrics (did the right chunk make the top five?) independently from answer metrics (faithfulness, completeness, correctness). Run it on every change to chunking, embeddings, prompts or the corpus itself. This is the same evaluation-first discipline we insist on for agents, applied one layer down — and in our experience it's the single change that turns RAG from a demo into a product.
Watch the long tail after launch
The questions users actually ask are not the ones in your test set. Log every query, retrieved passage set and answer; review the ones with low confidence, no citations or thumbs-down; and feed them back into the golden set and the ingestion backlog. Half of "the AI is wrong" tickets turn out to be "the document is missing or outdated" — which is a content-operations fix, not a model fix, and a much cheaper one.
The short version
RAG that survives production is mostly retrieval engineering: a clean, versioned corpus with rich metadata; structure-aware chunking; hybrid search with permissions and reranking; enforced grounding with citations; and separate evaluation of retrieval and generation, fed by real query logs. Get retrieval right and the model looks brilliant. Get it wrong and no model can save you.
Got a knowledge assistant that users stopped trusting? Tell us about it — it's usually fixable.