Retrieval-augmented generation (RAG): how to build it well
Retrieval-augmented generation (RAG) finds the relevant passages in your own documents at question time and hands them to a language model, which answers from them and cites its sources. It exists because models do not know your private or recent information. Most of the quality is decided by retrieval, so build and measure that part first.
What RAG is and why it exists
A language model knows what was in its training data up to a cut-off date. It has never seen your contracts, your product manuals or last week’s price list. Ask about them and it will often produce a fluent guess.
RAG fixes that without retraining anything. When a question arrives, the system searches your content, picks the passages most likely to contain the answer and puts them into the prompt next to the question. The model writes its answer from those passages and says which ones it used.
The name comes from a 2020 paper by Patrick Lewis and colleagues, which paired a text generator with a searchable index. The paper observed that models struggle to update what they know and to show where an answer came from. Those two points are still why businesses choose RAG: you change the system’s knowledge by changing the index, and a reader can check the source.
It is not the only option. A small set of documents may fit in the prompt with no retrieval at all, and a problem of behaviour is better solved by fine-tuning. We compare the three in RAG vs fine-tuning vs long context. This article assumes retrieval is the right choice and you want to build it properly.
One example runs through the rest of the piece: a distributor of industrial pumps with about 4,000 PDFs (manuals, service bulletins, parts catalogues, dealer price lists) and a support desk that answers questions like “what is the torque for the impeller nut on the P-340?”.
How does a RAG pipeline work?
There are two flows. Ingestion runs whenever documents change. The query flow runs for every question.
| Stage | What happens | What usually goes wrong |
|---|---|---|
| Ingest | Collect documents and extract text, tables and headings | Scanned pages, tables flattened into word soup, stale copies |
| Chunk | Split documents into passages small enough to retrieve precisely | A split in the middle of a table or a procedure |
| Embed | Convert each chunk into a vector that represents its meaning | An embedding model that is weak on your language or jargon |
| Index | Store vectors, keywords, the text and metadata | No metadata, so nothing to filter on |
| Retrieve | Find candidate chunks for the question | The right chunk is not among the candidates |
| Rerank | Score candidates against the question and keep the best | Skipped, so the model reads near misses |
| Generate | The model answers from the passages it was given | It answers from memory or blends two versions |
| Cite | Attach each claim to the passage that supports it | A citation that does not say what the sentence says |
Ingestion is where most of the effort goes, and it is ordinary data engineering: connectors, parsing, deduplication, and a way to notice when a source document changes or disappears. The pump distributor’s parts catalogues are tables. If the parser turns them into a stream of numbers with no column headers, nothing downstream will rescue them. Scanned and form-heavy documents deserve their own extraction work before they go anywhere near an index.
The query flow is short by comparison. Embed the question, search, rerank, build a prompt containing the top passages with their source labels, and ask the model to answer only from them and to say so when they do not contain the answer.
The decisions that decide quality
Chunking
A chunk has to be small enough to be about one thing and large enough to make sense alone. Splitting every few hundred tokens regardless of structure is the default in most tutorials and the cause of many bad answers. Split on the document’s own structure instead: sections, procedures, table rows kept with their headers.
Then give each chunk its context back. A passage that says “tighten to 85 Nm” is useless if nothing says which pump it refers to. Prefix every chunk with the document title, the section path and the product model. Anthropic’s write-up on contextual retrieval goes further and has a model write a short situating sentence for each chunk. In their tests, doing this for both the embeddings and the keyword index cut failed retrievals by 49%, and by 67% once a reranker was added. Those are their numbers on their data, but the direction matches what we see.
Hybrid search
Vector search finds passages with a similar meaning. It is poor at exact strings: part numbers, error codes, clause references, product names. Keyword search (BM25) is the opposite. Support engineers type “P-340” and “E-17”, so the distributor needs both.
Run the two searches in parallel and merge the result lists. The usual method is reciprocal rank fusion, which combines results by their position in each list instead of trying to compare scores that sit on different scales. Microsoft’s explanation of RRF is a clear reference. Most search engines can do this now, and so can Postgres with extensions, which makes a separate vector database a choice and not a requirement. We go through that decision in vector database or pgvector.
Reranking
First-stage retrieval is tuned for speed over a large index. A reranker is a slower model that reads the question and each candidate together and scores how well the passage answers it. Retrieve 50 to 100 candidates, rerank them, and pass the best 5 to 20 to the model. It adds some latency and a small cost per query, and after hybrid search it is usually the cheapest quality gain available.
Metadata filters
Store facts about each chunk as fields: product model, document type, version, date, language, region. Then filter before you search. A question about the P-340 should not retrieve the P-320 manual, however similar the wording, and a superseded service bulletin should lose to the current one. Without version metadata the model will happily merge two revisions into an answer that matches neither.
Permissions
Dealer price lists are not for every user. Enforce access in the retrieval query, using the permissions of the person asking, so a restricted passage never reaches the prompt. Asking the model to keep secrets is not access control. OWASP covers this under vector and embedding weaknesses and recommends permission-aware stores with strict partitioning. Plan it at the start. Keeping access rights in step with the source systems is harder to retrofit than any other part of the pipeline.
Why does RAG give wrong answers?
Every wrong answer is one of two things. A retrieval miss means the right passage never reached the model. A generation miss means it did, and the model still got it wrong. They need different fixes, and teams lose weeks tuning prompts for what is really a search problem.
For each failed question, look at what was retrieved and walk down this list:
- Is the answer in your documents at all? If not, the correct output is “I could not find this”, and the failure is that the system answered anyway.
- Is the passage in the index, intact? If the parser dropped the table, fix ingestion.
- Was it among the candidates? If not, look for a vocabulary mismatch (add keyword search), chunks without context (add headers) or a filter that is too strict.
- Did it survive reranking and the cut-off? If it sat at position 30 and you pass 10, change the reranker or the number of passages.
- Was it in the prompt and still ignored or misread? Now it is a generation problem: conflicting passages, an instruction that is too loose, a question that needs several passages combined, or a model too small for the reasoning.
In the builds we run, most failures in the first month fall under points 2 to 4. That is why logging the retrieved passages with every answer is not optional. Without them you cannot tell the two kinds of miss apart.
How do you evaluate a RAG system?
Build a test set before you tune anything. Take 50 to 150 real questions from the people who will use the system, and for each one record the passage that answers it and a short reference answer. Include questions with no answer in the documents, and questions whose answer the asker is not allowed to see.
Measure the two halves separately. For retrieval: how often is the right passage in the top results, and how high does it rank? For answers: is every claim supported by the retrieved passages, does the answer address the question, does it agree with the reference, do the citations point to the right place, and does the system decline when it should? Open-source tools such as Ragas implement these under names like context precision, context recall, faithfulness and response relevancy.
Several of those metrics use a model as the judge. Check the judge against human ratings on a sample before you trust its scores. Then re-run the whole set on every change to the chunking, the index, the prompt or the model, and keep sampling real conversations once the system is live. A score that only exists from launch week tells you nothing in month six, when half the documents have been replaced.
What should a first version include?
Our list for version one is short and unglamorous. One bounded set of documents that somebody owns. Parsing you have inspected by eye. Structure-aware chunks with context headers. Hybrid search and a reranker. Filters for product, version and date. Permission filtering. Citations the user can click. A clear “not found” response. The test set, and logs of question, retrieved passages and answer.
Leave out, for now: knowledge graphs, agents that plan several rounds of retrieval, fine-tuned embedding models, image and audio retrieval, and conversational memory beyond the last few turns. Some of these are worth adding once the test set shows a specific gap, such as questions that need facts from five documents at once. We describe when in agentic RAG and GraphRAG. Adding them before the basics are measured only makes failures harder to trace.
Also resist connecting every document source on day one. Ten thousand unowned, contradictory files make a worse assistant than four hundred current ones, and the people who pay for that are the users who stop trusting the answers.
How to start: the first week
Start by asking whether you need retrieval at all. Anthropic’s guidance in the article linked above is that a knowledge base under roughly 200,000 tokens, about 500 pages, can simply be placed in the prompt. If yours is larger, changes often or carries access rules, carry on.
Days one and two: collect 50 real questions with their answers and source pages. This is the most valuable thing you will produce all week. Day three: ingest a few hundred representative documents and read the parsed output yourself. Day four: set up hybrid search with a reranker and measure retrieval on the test set before writing any answer prompt. Day five: add generation with citations, run the full set, and sort every failure into retrieval miss or generation miss.
By Friday you will know whether the hard part is your documents, your search or your prompt, and that tells you what the next month will cost. If you would like a second pair of hands, this is the work our generative AI and RAG team does.