FinOps for AI: keeping GPU and inference bills under control
AI bills are driven by tokens and GPU hours, which move with every prompt change, traffic spike and idle endpoint, so a monthly cloud review catches problems too late. Keep them under control by measuring cost per use case, tagging every request to a feature and a customer, setting budgets with alerts on the rate of spend, and moving from API inference to your own GPUs only when steady utilisation makes the sums work.
Why does AI spend behave differently from cloud spend?
An ordinary cloud bill grows with the infrastructure you provision, and changes when someone provisions more. An AI bill grows with what users and prompts do, minute by minute.
The FinOps Foundation’s FinOps for AI overview, updated in October 2026, is a good map of the differences. Its starting point is reassuring: cost still equals price times quantity, and tagging, allocation and commitments still apply. What changes is the detail. The units are tokens, API calls and provisioned throughput. The tokens billed exceed what the user typed, because system prompts, conversation history and retrieved content are billed too. Prices change often. Provider bills frequently lack the tags you rely on for allocation. And the choice of model is a cost decision and a quality decision at once.
Its companion paper on how token pricing really works adds the mechanics. Output tokens often cost three to five times as much as input tokens. Model APIs are stateless, so the whole conversation is sent again on every turn and a session’s cost grows faster than its length. Reasoning tokens are often billed as output even when you never see them. Agents multiply all of this by making many calls per request.
This is now mainstream finance work. In the State of FinOps 2026 survey, 98% of respondents said they manage AI spend, up from 63% in 2025 and 31% in 2024.
Three practical consequences follow. A prompt edit is a pricing event. An agent stuck in a loop is a billing incident. And a review held once a month is too slow to catch either.
API inference or self-hosted GPUs?
| API, pay per token | Provisioned throughput | Self-hosted on GPUs | |
|---|---|---|---|
| You pay for | Tokens used | Reserved capacity for a term | GPU hours, busy or idle |
| Fits | Variable or growing traffic; the largest models; small teams | Steady high volume that needs latency guarantees | Steady load on an open-weight model; strict data location; small specialised models |
| Main risk | The bill scales with usage and prompt size | Paying for idle capacity; commitment to one model | Low utilisation; engineering time; getting capacity at all |
| Effort to run | Low | Low to medium | High |
The break-even question
Self-hosting has one sum that matters:
cost per million tokens =
GPU cost per hour
÷ (millions of tokens served per hour at full load × average utilisation)
Here it is with invented round numbers, chosen to show the shape and not to quote anyone’s prices. Suppose a GPU instance costs $2.00 an hour and your load test shows it sustaining 1,500 tokens a second across concurrent requests. At full load that is 5.4 million tokens an hour, or about $0.37 per million. At 20% average utilisation, busy in office hours and quiet at night, it is $1.85. At 5%, which is plausible for an internal tool used a few hours a week, it is about $7.40.
Utilisation moves the answer by a factor of twenty. That is the whole argument. An API charges nothing for the hours nobody is asking.
Then add what the API price already includes: engineers to run the serving stack, autoscaling, upgrades, on-call, and a second GPU for redundancy that doubles your idle time. Compare like with like as well. The model you can host may be smaller than the one behind the API, so test it on your own evaluations first. If a small or fine-tuned model passes, that is where self-hosting wins.
Measure throughput yourself. The FinOps Foundation’s paper on GenAI capacity options calls vendor tokens-per-minute figures a rough estimate and recommends load-testing with your real mix of input and output. The same paper treats provisioned capacity as a strategic commitment, not a discount: in its worked examples it cost more per token than pay-as-you-go even at full utilisation, and was justified by latency and uptime guarantees.
Our advice is to start on an API. Revisit when one workload has held a steady volume for three months, or when a requirement such as data location forces the question.
Tag every request to a feature and a customer
A provider invoice tells you how many tokens of which model you used. It does not tell you which feature, customer or release used them. You build that yourself, in the application or in a gateway that all model calls pass through. One record per call:
{
"ts": "2026-10-29T09:42:17Z",
"feature": "ticket-summary",
"customer_id": "c_1842",
"environment": "production",
"release": "2026-10-29.1",
"model": "<model-id>",
"input_tokens": 6120,
"cached_input_tokens": 4800,
"output_tokens": 310,
"outcome": "accepted"
}
Join that to a price table you keep current and you have cost by feature, customer and release. Use separate API keys or projects per environment and product as well, so the provider’s own usage reports split at least that far.
Report cost per outcome, not per token: per resolved ticket, per processed document, per qualified lead. Report it as a range, median and 95th percentile, because a few long agent runs hide behind an average. The FinOps Foundation overview makes the same recommendation.
Per-customer cost is how you find out whether a flat-priced plan has customers who cost more to serve than they pay.
Billing standards are catching up. FOCUS, the common format for cloud billing data, reached version 1.4 in June 2026, and AI model identity and token consumption are scoped for 1.5. Until billing exports carry that, the tagging is yours to do.
Budgets, alerts and runaway protection
Give each feature and environment a budget and an owner. Alert on the daily rate of spend against forecast, not on the monthly total, and on cost per request by release.
Here is why. A product team adds 3,000 tokens of instructions to the system prompt of an assistant that handles 40,000 requests a day. That is 120 million extra input tokens a day from one merged pull request. Nothing breaks. No alert fires unless you built one. Prompt caching softens the blow when the added text sits in a stable prefix, which is one of the techniques in cutting LLM costs.
Put the hard limits in code, not in a policy document:
- a maximum output length on every call;
- a maximum number of steps and a spending cap for each agent run;
- rate limits per user and per customer;
- provider-side monthly limits where they exist;
- development and CI keys with low caps of their own.
Right-sizing and autoscaling GPU workloads
If you do run GPUs, three habits carry most of the saving.
Right-size. Use the smallest GPU that fits the model at the precision you need. Quantised weights often allow a cheaper card, so test that on your evaluations. Look at actual GPU utilisation and memory, not at whether the instance is up. Serve with batching so that one GPU handles many requests at once. Throughput per GPU is the denominator in every sum above.
Autoscale on the right signal. Queue depth or requests in flight, not CPU. Scaling up is slow because model weights have to load, so keep a small warm floor for interactive traffic and scale batch and internal tools to zero. Check what your platform bills for that warm minimum. It is usually the full rate.
Hunt idle endpoints. This is the commonest waste we see: an endpoint deployed for a demo, a notebook with a GPU attached, a staging copy of production running through the weekend. Give every GPU resource an owner tag and an expiry date, and shut non-production down on a schedule.
Spot, on-demand or reserved?
Use interruptible capacity for work that can restart: training, fine-tuning, batch inference and evaluation runs. On AWS a Spot Instance gets a two-minute warning before it is stopped, so checkpoint often. Do not put an interactive endpoint on spot without on-demand capacity behind it.
Reserve only a baseline you have measured for two or three months. GPU commitments age quickly, and a year’s commitment to a card your next model does not fit on is waste. Cover the floor with commitments, the peaks with on-demand and the restartable work with spot.
The hidden lines on the bill
- Egress. Documents sent to a model in another cloud, or model weights pulled repeatedly from a registry in another region.
- Vector storage and re-embedding. Indexes want memory, and a change of embedding model means re-embedding everything. See how to choose a vector store.
- Logging and tracing. Storing every prompt and response is useful and bulky, and observability tools charge by volume. Set retention deliberately, as described in LLMOps in production.
- Evaluation. Running the test suite on every change costs tokens, judge-model calls included. It is worth it. Budget for it.
- Retries and fallbacks. A timeout that retries the whole request, or a fallback to a dearer model.
- Seats and scattered subscriptions. AI tools bought on a card by teams outside engineering.
A monthly review checklist
Forty-five minutes with an engineer who can explain each line, the product owner and someone from finance.
- Total AI spend against budget, split into API tokens, GPU compute and everything else.
- Cost per outcome for each feature, median and 95th percentile, against last month.
- The ten most expensive customers to serve, against what they pay.
- Token mix per feature. Has average input per request crept up?
- Cache hit rate, and the share of non-urgent work going through batch pricing.
- GPU utilisation per endpoint. Anything under your threshold is resized, scheduled or removed.
- Idle and orphaned resources: endpoints with no traffic, old indexes and snapshots.
- Commitment coverage and utilisation, and anything expiring in the next 90 days.
- The month’s anomalies, and whether an alert caught each one.
- Provider changes: a cheaper model that passes your evaluations, or a retirement that forces a migration.
Every item ends with an owner and a date, or it was only a discussion.
How to start
In the first week, do four things. Add the per-request record above to every model call. Build one dashboard showing cost by feature and by day. Set one alert on daily spend. List every running GPU endpoint with its owner, and switch off the ones nobody claims.
The first decision is the unit you will manage to: cost per what? Pick the outcome the business already counts. Cost visibility of this kind is part of the cloud, FinOps and LLMOps work we do.