Cutting LLM costs: caching, routing, batching and small models
The biggest savings on an LLM bill come from sending fewer tokens and paying less for the ones you repeat, not from switching provider. In order, we would measure cost per feature, trim the context, turn on prompt caching, route easy requests to a cheaper model, batch anything that is not urgent, cap output length, and only then consider fine-tuned small models or self-hosting.
Measure first, then work down this list
An LLM bill is tokens multiplied by price, summed over every call. Three kinds of token are priced differently. Input tokens are what you send. Output tokens are what the model writes, and on the main providers’ price lists they cost several times more per token than input. Cached input tokens are repeated input that the provider has seen recently, billed at a small fraction of the normal rate. If the model reasons before answering, that hidden reasoning is billed as output too.
Most teams know their total and nothing else. Before changing anything, tag every call with the feature that made it, and log input, cached and output tokens per call. A week of that data usually shows two things: one or two features account for most of the spend, and input tokens dominate, because the same instructions, history and documents are resent on every request.
Take a generic support assistant that answers 20,000 questions a day. Each request carries a 3,000-token system prompt, 2,000 tokens of tool definitions, 6,000 tokens of retrieved articles and 4,000 tokens of chat history, and gets a 300-token reply. That is 15,000 tokens in and 300 out. Suppose output is priced at five times input. The reply then costs the same as 1,500 input tokens, about a tenth of the call. Shortening the reply would save almost nothing. The money is in the input.
This is the order we would work in, with the cheapest and safest changes first.
| Order | Step | What it saves | Effort | Risk to quality |
|---|---|---|---|---|
| 1 | Measure cost per feature | Nothing directly, it aims everything else | Low | None |
| 2 | Trim the context | Input tokens on every call | Low to medium | Low, often improves answers |
| 3 | Prompt caching | Most of the price of repeated input | Low | None |
| 4 | Route by difficulty | Price per token on easy requests | Medium | Medium, needs evaluation |
| 5 | Batch non-urgent work | About half the price of those jobs | Low | None, but answers are delayed |
| 6 | Control output and reasoning | Output tokens | Low | Low to medium |
| 7 | Smaller or fine-tuned models | Price per token on a whole task | High | Medium to high |
| 8 | Self-hosting | Depends entirely on utilisation | Very high | Operational |
Trim the context
Every token you send is paid for on every request, so the first job is to stop sending things the model does not need.
The usual finds are these. Retrieval that returns ten long chunks when three would do: add a reranker and cut the number. Chat history sent in full for ever: summarise older turns, and drop tool outputs once they have been used. System prompts that have grown by accretion, with rules for cases that no longer exist. Definitions for 40 tools when any one request can only need five.
In the support example, cutting retrieval from 6,000 to 2,500 tokens and history from 4,000 to 1,500 takes the request from 15,000 tokens to 9,000 with no change of model. Answers often improve at the same time, because the model has less irrelevant text to wade through. We go through the method in our article on context engineering.
Do this before you turn on caching. Caching makes repeated tokens cheap, which hides waste without removing it.
How does prompt caching work?
When a model processes a prompt, the provider can keep the processed form of the beginning of it for a while. If your next request starts with exactly the same content, that prefix is read from the cache: billed at a fraction of the normal input price and handled faster. Everything after the first point of difference is processed and billed as normal.
The three large providers implement the same idea with different controls. Check their pages for current rates, because these change.
- Anthropic. You opt in with a
cache_controlmarker, either once for the whole request or on specific blocks. The cache covers tools, then the system prompt, then messages, in that order. The default lifetime is five minutes, refreshed each time the entry is used, with a one-hour option. Writing to the cache costs more than normal input and reading from it costs far less. Details are in the prompt caching documentation. - OpenAI. Caching is on by default for supported models once a prompt passes a minimum length. On its newer models you can also set explicit breakpoints and cache lifetimes, and cache writes carry a premium. See the prompt caching guide.
- Google. Gemini applies implicit caching by default on current models and passes on the saving when a request hits the cache, with no guarantee that it will. Some earlier model generations also support explicit cache objects that you create and manage. See the caching documentation.
The rules that follow from the mechanism are the same everywhere.
Put stable content first and variable content last. A typical order:
1. Tool definitions (same for every user)
2. System instructions (same for every user)
3. Reference documents (same for many requests)
4. Conversation so far (grows, but only at the end)
5. The new user message (always different)
Keep the prefix identical, character for character. The classic mistake is a timestamp or the user’s name near the top of the system prompt, which makes every request unique and every cache lookup a miss. Another is building the tool list in a different order on each call.
Mind the minimum size and the lifetime. Short prompts are not cached at all, and the threshold varies by model. An entry that is not reused within its lifetime expires, so caching pays on busy features and does little for one that is called twice an hour. Where writes cost extra, a prefix has to be read at least a couple of times to come out ahead.
Then verify. Every provider reports cached tokens in the usage data of the response. Log it. If the cached share is near zero on a feature with a long fixed prompt, something in the prefix is changing.
Route requests by difficulty
Providers sell models in tiers, and the price gap between the cheapest and the most capable is wide. Many requests in a real product are easy: greetings, simple lookups, reformatting, yes-or-no classification. Sending those to the top model is waste.
There are three ways to route, in rising order of effort:
- By feature. The autocomplete uses the small model and the contract analysis uses the large one. There is no decision at run time. Start here.
- By rule or classifier. A cheap first step labels the request as simple or hard, using length, intent or which tools are involved.
- By cascade. The small model tries first and its answer is checked by a validation rule or a confidence score. Failures go to the larger model.
Routing is the first step on this list that can hurt quality, so it needs a test set. Take a few hundred real requests, run them through both models, and have a person or a strong model judge the pairs. Find the categories where the cheap model is as good, route those, and leave the rest.
One interaction to watch: a cache belongs to one model. If traffic is split thinly across several models, each sees fewer repeats of the prefix and the cached share falls.
Batch work that is not urgent
If nobody is waiting for the answer, do not pay for an immediate one. Batch interfaces accept a large set of requests, process them when capacity is free, and charge less.
At the time of writing, OpenAI’s Batch API, Anthropic’s Message Batches API and Google’s Gemini Batch API each price batch work at half the standard rate, with results expected within 24 hours and often much sooner. Anthropic notes that the batch and caching discounts can be combined. OpenAI also offers a flex tier, in beta, that gives batch-level pricing on ordinary calls in exchange for slower and occasionally refused responses.
Good candidates are nightly document processing, backfills, evaluation runs and report generation. In most products we look at, something runs synchronously only because that was the first way it was written.
The work is in the plumbing. You need to submit jobs, poll for completion or receive a webhook, match results back to records by ID, and handle the few requests that fail or expire.
Control output length and reasoning
Output tokens are the expensive ones, and they also set response time, because a model writes one token after another.
Ask for less. Tell the model the length and format you want, and set a maximum output limit as a backstop. Where the result feeds code, request a compact structured output instead of prose.
Reasoning deserves the same attention. Models that think before answering bill that thinking as output, and current APIs let you set how much effort they spend. A classification call seldom needs deep reasoning. Lower the setting on simple features and rerun the test set to confirm nothing got worse.
In an agent the saving compounds, since each step’s output becomes input to every later step.
Smaller models, fine-tuning and self-hosting
These are last on purpose. They can save the most on a single task, and they cost the most to do properly.
A small model fine-tuned on your task can replace a large general one for a narrow, high-volume job, with a much shorter prompt. The price is a labelled dataset, an evaluation process and a model you now have to maintain. The route has also narrowed: OpenAI is winding down its self-serve fine-tuning, so in practice this now means fine-tuning an open-weight model or using a cloud platform that still offers tuning. We cover when a small model is enough in small language models.
Self-hosting changes the billing from per token to per hour. A GPU costs the same idle or busy, so the price per answer is set by utilisation. It makes sense when volume is large and steady, when data rules require it, or when you already run GPU infrastructure for other reasons. For spiky or modest traffic, a small hosted model with caching and batching is usually cheaper. Our piece on FinOps for AI goes into the GPU side.
How to start: the first week
- Add a feature tag and token counts (input, cached, output) to every model call.
- After three or four days of data, rank features by spend. Pick the top one.
- Read ten real requests from that feature in full, exactly as sent. Mark what the model did not need.
- Trim it, reorder the prompt so the stable part comes first, and turn on caching. Check the cached share in the usage data.
- List every job that does not need an immediate answer and move the largest to a batch interface.
Those five steps involve no change of model and little risk to answer quality, and they are usually where most of the saving is. Routing and smaller models come after, with a test set in hand. If the bill is tied up with infrastructure choices as well, LLMOps and FinOps are part of the cloud work we do at WeStack AI.