BlogGenerative AI

RAG vs fine-tuning vs long context: choosing the right one

Retrieval-augmented generation (RAG) changes what a model knows at the moment it answers, fine-tuning changes how it behaves, and a long context window changes how much it can read in one request. Most teams should start with a clear prompt plus retrieval, put the material straight into the context when it is small enough to fit, and fine-tune only to fix behaviour or to cut cost at volume.

Three tools for three different problems

Teams often treat RAG, fine-tuning and long context as rival answers to one question: how do we make the model know our business. That framing causes most of the wasted months we see. The three change different things.

RAG changes what the model knows right now. Before the model answers, your system searches your own content, picks the passages that look relevant and places them in the prompt. The model’s weights are untouched. The idea dates from a 2020 paper by Lewis and colleagues, which paired a language model with a searchable index so that knowledge could live outside the model.

Fine-tuning changes how the model behaves. You train it further on examples of the input and the output you want, and the weights shift. It is good at teaching a habit: a house style, a strict output format, a narrow classification skill. It is poor at teaching facts that change.

Long context changes how much the model can read at once. The largest models from Anthropic and Google now accept around a million tokens in a single request, which is several thick books. If your material fits, you can skip the search step and hand over everything.

So the first question is a diagnostic one. Is the model failing because it lacks information, because it behaves the wrong way with the information it has, or because it only sees fragments of something it needs to read whole?

What does each approach cost you?

RAG

A RAG system has moving parts: document parsing, chunking, an embedding model, a vector index, usually a keyword index beside it, a reranker, and the prompt that stitches the results together. None of it is exotic, but each part can fail without raising an error. The work is mostly data engineering, which is why the state of your documents matters more than your choice of model. Our guide to RAG walks through the build.

Per request, RAG is cheap. You send a few thousand tokens of retrieved text instead of the whole library. The search step adds a little latency, usually small next to the time the model spends writing. Freshness is its best property: update the index and the next answer reflects it, with no retraining. It also gives you citations nearly for free, because you know exactly which passages were passed to the model and can show them.

Fine-tuning

Fine-tuning moves the cost to the front. You need examples, and they have to be good, because the model will copy your mistakes faithfully. For a narrow task we plan for a few hundred carefully checked examples as a starting point, plus a separate test set that the model never trains on. Then there is the training run, the evaluation, and a repeat whenever the base model is retired or your requirements shift.

The payoff comes at inference time. A tuned model needs a shorter prompt, since the instructions and examples are baked in, and a small tuned model can often do a narrow job that would otherwise need a large general one. That is where the cost argument comes from.

One thing changed during 2026 that affects planning. OpenAI has been winding down its self-serve fine-tuning platform: it is closed to new users, and its deprecations page says existing customers can no longer create new jobs from 6 January 2027. Google Cloud still documents supervised tuning for several Gemini models. In the builds we run, most fine-tuning is now done on open-weight models with LoRA, a method that freezes the original weights and trains a small set of added ones. You own the result and no vendor can retire it, but you also own the hosting. Our fine-tuning guide covers when that trade is worth making.

Long context

Long context has almost no build cost. There is no index and no pipeline. You pay on every request instead, in money and in time, because the model has to process everything you send. Prompt caching softens this a great deal when the same material is reused across requests, and Google’s documentation calls caching the primary optimisation for long-context work.

The less visible cost is accuracy. Anthropic’s documentation says plainly that accuracy and recall degrade as the token count grows, and it uses the name context rot for the effect. Google’s long-context guide makes a related point: models are good at finding one fact in a huge input and less good when they must find several. A million-token window is a capacity limit. It does not promise that the model will use every part of it equally well.

How do they compare side by side?

RAG Fine-tuning Long context
What it changes What the model knows for this question How the model behaves on every question How much the model reads in one request
Freshness Minutes, once the index updates Frozen at training time Immediate, you send the current text
Citations Natural, you know the passages used None, knowledge is blended into weights Possible, if you ask for quoted passages
Data needed Your documents, cleaned and parsed Labelled pairs of input and output Your documents, as they are
Upfront effort Medium: pipeline and evaluation High: dataset, training, evaluation Low
Cost per request Low Lowest, on a small tuned model High unless cached
Latency A search step plus a short prompt Fast Slowest for large inputs
Upkeep Index sync, retrieval quality checks Retrain on drift or model retirement Watch the size as content grows
Typical failure The right passage is not retrieved Stale facts stated with confidence Detail lost in a very long input

Which one should you choose?

Start from the symptom.

If the problem is Start with Why
The model does not know your products, policies or records RAG Knowledge lives outside the model and stays current
The content is small and stable, up to a few hundred pages Long context with caching No pipeline to build or debug
Answers must show their sources RAG The passages are known and can be cited
Format or tone keeps drifting despite a clear prompt Fine-tuning Behaviour is what training changes
A narrow, high-volume task costs too much on a large model Fine-tune a small model Shorter prompts on a cheaper model
The model must reason across one long document Long context Chunking would break the argument apart
Different users may see different documents RAG with access filters Permissions are enforced at retrieval

On the second row, Anthropic’s retrieval guidance has said since 2024 that a knowledge base under roughly 200,000 tokens, about 500 pages, can simply go in the prompt. Windows have grown since then, but we would not stretch the rule in proportion. Cost per request and loss of accuracy both grow with size.

Our plain recommendation: begin with a well-written prompt and retrieval. It covers the largest share of real business problems, it is the easiest to debug, and it does not lock you in. Reach for fine-tuning when you have evidence of a behaviour problem that prompting cannot fix, or a cost problem at volume.

Four scenarios

A support assistant over 4,000 help articles. The articles change every week and customers want a link to the source. This is RAG. Fine-tuning would freeze last month’s answers into the weights, and the library is too large and too changeable to send whole.

An HR assistant for a 60-page staff handbook. The handbook changes twice a year. Put the whole thing in the context and cache it. Building a retrieval pipeline for 60 pages is effort spent on a problem you do not have.

Sorting 200,000 inbound messages a day into 30 categories. The categories are fixed and the output is a short JSON object. A large model with a long instruction prompt does the job, and the bill is painful. Collect a few thousand labelled messages from the system you already run, tune a small model on them, and keep the large model for the messages the small one is unsure about.

Reviewing a 150-page supply contract against a negotiation playbook. The contract must be read whole, because clause 14 changes the meaning of clause 3. That is long context. The playbook and the past positions run to thousands of pages, so those are retrieved. Mature systems mix the approaches in this way.

Where does each approach break?

RAG fails at retrieval far more often than at generation. The answer was in your documents and the search did not surface it, perhaps because the question used different words, or the answer was split across two chunks, or it needed facts from three documents at once. Single-shot search also cannot answer questions about the collection as a whole, such as what customers complained about most last quarter. When you hit those limits, the next step is agentic retrieval or a knowledge graph. Fine-tuning will not help with either.

Fine-tuning fails slowly. The model is excellent on launch day and drifts out of date as products and policies change. Because the knowledge is in the weights, it cannot say where an answer came from, and it will state an outdated fact with full confidence. We also see teams fine-tune before anyone has written a test set, which leaves nobody able to say whether the tuned model is better.

Long context fails by dilution. Teams keep adding material because the window allows it. The bill and the response time climb, and answers get worse as the relevant paragraph is buried under irrelevant ones. Deciding what goes into the window and what stays out is a discipline of its own.

How to start: the first week

This is a week of work, about the size of a small proof of concept.

  1. Write down 30 to 50 real questions or tasks, each with the answer you would accept. This is your test set, and every later decision is judged against it.
  2. Count your tokens. If all the source material fits comfortably in the model’s window, start with long context and caching, and skip step 3.
  3. Otherwise build the plainest retrieval you can: sensible chunks, keyword and vector search together, the top results in the prompt.
  4. Run the test set. For each failure, record whether the right passage was retrieved. If it was not, fix retrieval. If it was and the answer is still wrong, fix the prompt.
  5. Only after that, look at what remains. Failures of format or tone that survive several prompt changes are candidates for fine-tuning. So is a cost line dominated by one narrow, repetitive task.

Most projects reach step 5 with no reason to fine-tune, and that is a good outcome. If you want a second pair of eyes on the diagnosis, this is the kind of generative AI and RAG work we do at WeStack AI.

Questions

Asked often.

Is RAG better than fine-tuning?

They do different jobs. RAG gives a model facts it can look up at the moment of the question, so it suits knowledge that changes and answers that need sources. Fine-tuning changes how a model behaves: tone, format, a narrow skill. If the problem is missing or stale knowledge, use retrieval. If it is inconsistent behaviour, consider fine-tuning.

Do long context windows make RAG unnecessary?

For a small, stable body of text, yes: put all of it in the prompt and cache it. For anything large or fast-changing, or where users hold different permissions, no. Every extra token costs money and time on each request, and models become less reliable at finding details as the input grows. Retrieval remains the way to send only what matters.

How much data do you need to fine-tune a model?

It depends on the task, and quality matters more than volume. For a narrow job such as classifying tickets or writing in a fixed format, we plan for a few hundred carefully checked examples as a starting point, plus a held-out test set. If you cannot assemble that, fine-tuning is premature and prompting with examples will serve you better.

Can you combine RAG, fine-tuning and long context?

Yes, and mature systems usually do. A common shape is retrieval to fetch the facts, a long window so the model can read whole documents instead of fragments, and a small fine-tuned model for a high-volume step such as routing or extraction. Start with one, measure, and add another only when a specific failure calls for it.

Not on the list? Ask us directly

Keep reading

More notes from the build.

All blogs

Let’s build your stack.

Send a brief and we’ll reply with next steps.