Fine-tuning LLMs: when it pays off and how to do it
Fine-tuning pays off when you need a model to behave in one specific, consistent way on a narrow task: a fixed format, a house tone, or a small model doing a large model’s job at lower cost and latency. It changes behaviour, not knowledge, so it is the wrong tool for facts that change. Start only when you have an evaluation set and a prompt-only baseline to beat.
What does fine-tuning change?
Fine-tuning continues the training of an existing model on your own examples. You show it several hundred or several thousand inputs, each paired with the output you wanted, and its weights shift towards producing that kind of output.
What it learns is how to respond: the format, the tone, where a decision boundary sits, what to leave out. What it does not learn in any dependable way is a body of facts. Information absorbed into weights is recalled approximately. It cannot be cited, and it cannot be corrected without training again.
That is the distinction the rest of this article hangs on: fine-tuning changes behaviour, not knowledge. If the gap is knowledge, use retrieval-augmented generation. If you are still choosing between approaches, read RAG vs fine-tuning vs long context first.
When does fine-tuning pay off?
The good reasons are few and specific.
Consistent format or tone. A prompt gets the structure right most of the time and you need it right nearly every time, or your brand voice survives the first paragraph and then drifts.
A narrow task at lower cost and latency. A long prompt full of rules and examples can be replaced by a short prompt on a tuned model. Google’s Vertex AI tuning documentation lists exactly this, cutting context by removing few-shot examples, alongside classification, fixed-format summarisation and extraction.
A smaller model matching a larger one. If a large model already does the task well, its outputs can train a small one to do the same job at a much lower running cost. It also makes it realistic to run the model privately, on your own hardware.
Tasks that are easier to show than to describe. Some judgements resist a written rule but are obvious from 500 examples.
The bad reasons are more common:
- Teaching facts that change, such as prices, policies or a product catalogue.
- Avoiding prompt work. If a strong model with a careful prompt and good examples cannot do the task, your examples are probably inconsistent, and training on them will not help.
- Wanting “our own model” as an end in itself.
- Low volume. With a few hundred requests a day there is nothing to pay the effort back.
Here is the scenario we will use. An online marketplace receives two million new listings a month and must place each one in a taxonomy of 900 categories and extract a dozen attributes. A large model with a prompt containing the whole taxonomy and worked examples does this well, but every listing carries thousands of prompt tokens and the response is slow. The task is narrow and stable, the volume is high, and every answer can be scored. That is a fine-tuning candidate.
The same company’s seller help assistant, which answers questions about fee policies that change every quarter, is not.
The methods in plain words
| Method | What gets trained | What it needs | When we reach for it |
|---|---|---|---|
| Full fine-tuning | Every weight in the model | The most data and GPU memory; the result is a complete copy of the model | An open-weight model where adapters fell short |
| LoRA and other adapter methods | A small set of added weights; the base model stays frozen | Much less memory; the result is a small adapter file | The default for almost every business task |
| Distillation | A small “student” model, on outputs from a large “teacher” | Plenty of real inputs; the teacher supplies the labels | A large model already does the job and you want it cheaper |
LoRA is the method worth understanding. The original paper froze the pretrained weights and trained small low-rank matrices alongside them. On the authors’ comparison it cut the number of trainable parameters by a factor of 10,000 and GPU memory by a factor of three, with no extra delay at inference. Variants such as QLoRA train against a compressed copy of the base model so the job fits on smaller hardware.
Adapters sit underneath several hosted services too. Google’s tuning exposes an adapter size setting, and LoRA is the default on Together AI.
Distillation is less a training method than a way of getting labels. You run the large model over real inputs, keep its outputs after checking a sample, and tune the small model on them. Amazon Bedrock offers this as a managed workflow that can draw on your production logs. For the marketplace this is the natural route, because the large model has already labelled millions of listings.
How much data, and how good?
Quality first. OpenAI’s fine-tuning guide suggests beginning with about 50 well-made examples and checking the result before collecting more. Our planning figures are a few hundred clean examples for a format or tone task and a few thousand for classification with many classes, because each of the marketplace’s 900 categories needs enough examples to be learnt, including the rare ones.
What makes a dataset good:
- Inputs look like production. Same prompt template, same mess, same languages.
- Labels are consistent. If two of your own people disagree on an example, the model will learn the disagreement.
- Edge cases are present, including “none of these” and inputs the model should decline.
- A test set is held back and never used in training, with near-duplicates removed so the score is honest.
- Personal data is removed or justified. Models can memorise what they are trained on.
Many services take training data as JSON Lines, one example per line, in the same chat format you use at inference. For the marketplace, one line looks like this (wrapped here for reading):
{"messages": [
{"role": "system", "content": "Classify the listing. Return category_id and attributes as JSON."},
{"role": "user", "content": "Title: 18V cordless drill with two batteries, boxed\nDescription: Used twice, charger included."},
{"role": "assistant", "content": "{\"category_id\": \"tools.power.drills\", \"attributes\": {\"voltage\": \"18V\", \"condition\": \"used\"}}"}
]}
The system message should be the one you will run in production, word for word. Machine-generated examples are a reasonable way to start, provided a person reviews a sample before you train on them.
The workflow, from baseline to monitoring
- Baseline. Write the best prompt you can and score it on the evaluation set, on both a large model and the small one you hope to tune. This is the number to beat. Quite often it ends the project, because the prompt turns out to be enough. Our prompt engineering guide covers that step.
- Dataset. Collect, clean, label and split into training, validation and test.
- Train. Begin with the service’s defaults and an adapter. Change one setting at a time and watch the validation loss for overfitting.
- Evaluate against the baseline. Same test set, same scoring. An overall accuracy figure hides the rare categories, so score per category and read the worst twenty. Also check what you did not train for: does the tuned model still follow system instructions and still decline what it should?
- Deploy carefully. Run the tuned model in shadow beside the current one, compare on live traffic, then switch.
- Monitor. Sample and score outputs every week. Inputs drift: new product types appear, the taxonomy changes, and accuracy slides without anyone touching the model.
OpenAI’s guide puts the first step bluntly: only invest in fine-tuning after setting up evals.
Hosted fine-tuning or open-weight models?
The hosted options have changed a good deal, so this is the position at the time of writing.
- OpenAI is winding down its fine-tuning platform. It is closed to new users, existing customers can create training jobs until 6 January 2027, and tuned models keep working only until their base model is shut down.
- Anthropic states in its documentation that the Claude API does not currently offer fine-tuning.
- Google offers supervised fine-tuning on Vertex AI for several Gemini Flash, Flash-Lite and Pro models.
- Amazon Bedrock supports fine-tuning for Amazon’s Nova models and several Meta Llama models, plus the distillation workflow above.
- Open-weight specialists such as Together AI offer LoRA and full fine-tuning on open models and let you download the adapter or the merged weights.
Our reading: tuning the largest closed models is turning into a by-arrangement service, and the dependable place to fine-tune is a small or mid-sized model whose weights you can hold.
Hosted tuning on Vertex AI or Bedrock is the simplest route if you already run on that cloud and accept the provider’s model lifecycle. Open weights give you control and portability. In exchange you take on serving: GPUs, scaling, and paying for capacity when traffic is quiet. That is MLOps work, and it belongs in the budget from the start.
What does it cost, and what happens when the base model retires?
In order of size, the costs usually run like this. Data comes first: people’s time to label, review and argue about edge cases. Then the evaluation set and the tooling to run it, which you need anyway. Training compute is often the smallest line, particularly with adapters on a small model.
Hosting depends on the route. A hosted service may charge per token. Your own deployment, or a dedicated endpoint, bills by the hour whether or not requests arrive. Together AI, for example, serves tuned adapters on dedicated endpoints only.
Then there is maintenance, which is where plans go wrong. A tuned model is tied to its base. When a hosted base model is retired, every model tuned on it stops working on the same day, and an adapter cannot be carried across to the successor. You re-tune and re-evaluate. Anyone who built on OpenAI’s fine-tuning now has that date in the calendar.
So treat the dataset, the training configuration and the evaluation set as the asset, and keep them in version control. The weights are replaceable. With that in place, moving to a new base model is a planned job instead of an emergency.
Finally, check the sums against the alternatives. Caching, routing easy requests to a cheaper model and trimming the prompt often recover much of the saving with no training at all; see cutting LLM costs.
How to start
Spend the first week on measurement, not training. Write 200 evaluation examples from real traffic. Score your best prompt on the large model you use today and on two small candidates.
Then make one decision. If a small model with a good prompt is already close to your target, stop: you do not need to fine-tune. If the gap is consistency or format, tune an adapter on a few hundred examples and give the experiment a two-week limit. If the gap is knowledge, build retrieval instead.
Whichever way it goes, you finish the week with an evaluation set you will use for years. If you want help running that first comparison, it is part of the LLM development and fine-tuning work we do.