# LLMOps: monitoring, versioning and evaluating LLM apps

> What LLMOps adds to normal operations: versioning prompts, models and tools as one release, tracing every request, and evals in CI and on live traffic.

- Published: 2026-07-23; last updated 2026-11-13
- Author: the WeStack AI team
- Subject: MLOps
- Address: https://westack.ai/blog/llmops-in-production

> LLMOps is the practice of running an LLM application so that you know what it did, can tell when its quality changes, and can change it safely. In practice that means versioning the prompt, model, tools and retrieval settings as one release, tracing every request, running evaluations before each change and on a sample of live traffic, and having a plan for the day your provider retires the model.

## How is LLMOps different from MLOps and normal operations?

Normal software operations assume that the same input gives the same output and that behaviour changes only when you deploy. Monitoring watches errors and latency. If the service answers quickly with a 200, it is healthy.

MLOps added models you train yourself. Its concerns are training pipelines, feature stores, data drift and retraining schedules.

LLM applications break assumptions from both.

- **You do not own the model.** A provider trains it, updates it and retires it on their timetable.
- **Behaviour is configuration.** The prompt, the retrieved context, the tool definitions and the model settings decide what the system does. A one-line prompt edit can change behaviour as much as a code release, and it often ships with far less ceremony.
- **Outputs vary.** There is rarely one correct answer, so testing means scoring against criteria, not checking equality.
- **Failures look like successes.** A wrong answer arrives on time with a 200 status. Conventional monitoring cannot see it.
- **Cost per request is variable.** It moves with context length and the number of steps an agent takes.

So the practice comes down to three abilities: knowing exactly what ran, seeing what it did, and measuring whether that was good.

## Version prompts, models and tools together

The unit of release is a bundle, not a prompt. We write it down as a manifest.

```yaml
release: support-assistant-2026-07-23.2
model:
  id: <pinned-model-id>        # a fixed version, never a moving alias
  max_output_tokens: 1024
prompts:
  system: prompts/system.md@4f2a9c1
  escalation: prompts/escalation.md@b81d0e7
tools:
  - name: lookup_order
    schema_version: 3
  - name: create_refund
    schema_version: 5
retrieval:
  embedding_model: <embedding-model-id>
  index_version: kb-2026-07-23
  top_k: 8
guardrails: policies/v7.yaml
evals:
  suite: golden-v12
  result: pass
```

Four rules follow from it.

1. Every part is in version control and referenced by something that cannot change underneath you: a commit hash, a schema version, a pinned model ID.
2. The release ID is attached to every trace, so any output can be tied to the exact configuration that produced it.
3. Rollback restores the whole bundle. A prompt tuned for one model may behave badly on another.
4. The knowledge index is a version too. Re-indexing documents changes behaviour without touching code.

Prompt management tools that let product people edit prompts are fine, provided an edit creates a new version that passes the same evaluations as a code change. A prompt edited live in a dashboard is an untested deployment.

The part that breaks most quietly is the tools. Another team changes the response of an internal API, nothing in your repository changes, and the agent starts working from a field that no longer exists. Version the tool schemas, and add a contract test that calls each real tool and checks the shape of what comes back.

## Trace every request

A trace is the tree of steps behind one request: the incoming message, the retrieval, each model call, each tool call, the final response. Without it, debugging is guesswork. You cannot tell a retrieval failure (the right document never arrived) from a generation failure (it arrived and the model ignored it).

| What to log | Why |
| --- | --- |
| Release ID | Ties the output to the exact prompt, model, tools and index |
| Model ID reported in the response | Shows when an alias has moved to a new version |
| Input, cached and output tokens | Cost per request, feature and customer |
| Latency per step and time to first token | Finds the slow step, not just the slow request |
| Retrieved document IDs and scores | Separates retrieval failures from generation failures |
| Tool calls with arguments, results and errors | Agents fail in their tools more often than in their prose |
| Prompt and response text, under your retention policy | Reproducing a problem and building test cases |
| Customer, session and feature tags | Cost allocation and investigating a complaint |
| Finish reason and guardrail events | Truncated outputs, refusals and blocked content |
| User feedback and what happened next | The only direct signal of quality |

Record the metadata for every request. Text is the expensive and sensitive part, so capture it for a sample, or for a limited period, if volume or privacy rules require.

### Open standards for tracing

OpenTelemetry has semantic conventions for generative AI: agreed names for the spans and attributes (the `gen_ai.*` family) that describe model calls, token usage, tool execution and agent steps. Their current state is worth knowing precisely. They now live in [their own repository](https://github.com/open-telemetry/semantic-conventions-genai), separate from the main semantic conventions, and the specification is still marked Development, which means names can change.

Two things follow. Instrument with OpenTelemetry-compatible libraries anyway, because it keeps you portable: backends such as Langfuse [accept traces over OTLP](https://langfuse.com/docs/opentelemetry/get-started) and map the `gen_ai` attributes to their own model. And expect renames. Pin your instrumentation versions and treat the attribute names in your dashboards as something that needs upkeep.

The conventions also make recording message content opt-in. By default an instrumentation should not capture prompts and responses. That is the right default, and it leaves the retention decision with you.

## Evals in CI and on live traffic

Before release, keep a golden set of real cases with the behaviour you expect, and run it on every change to anything in the bundle. Block the merge when the score falls beyond a tolerance you have agreed. We use three kinds of check, in order of preference:

1. **Deterministic assertions.** The output is valid JSON, it includes the order number, the right tool was called with the right arguments.
2. **Model-graded rubrics.** A judge model scores the output against written criteria. Calibrate the judge against human labels on a sample, and again whenever the judge model changes.
3. **Human review.** A small sample on each release, read by someone who knows the domain.

After release, score a sample of live traces with the same rubrics, asynchronously. Watch the user signals: thumbs-down, rephrased questions, escalations to a person. Send new releases to a slice of traffic first and compare by release ID.

Every production failure you investigate becomes a new golden case. A set that never grows ends up testing last quarter's traffic. And because outputs vary between runs, run the important cases several times and compare distributions, or one unlucky sample will block a good release. For [agents that call tools](https://westack.ai/services/ai-agents), assert on the tool calls as well as the text. The method is covered in more depth in [how to evaluate AI agents](https://westack.ai/blog/how-to-evaluate-ai-agents).

## What happens when the provider retires a model?

It will happen, and the notice periods are published. Anthropic commits to [at least 60 days' notice](https://platform.claude.com/docs/en/about-claude/model-deprecations) before retiring a publicly released model, after which requests to it fail. OpenAI's policy is [at least six months](https://developers.openai.com/api/docs/deprecations) for generally available models and at least three for specialised variants, while preview models may go with as little as two weeks. Request parameters are retired as well as models, and cloud platforms that resell a model can set their own dates.

What we do about it:

1. One named person owns the model lifecycle. Retirement dates go in a shared calendar and provider emails go to a team mailbox.
2. No preview models on anything the business depends on.
3. Migration is an evaluation exercise. Run the golden set against the candidate, expect to adjust prompts, check cost and latency as well as quality, then send it a slice of traffic.
4. Start early. Two models a year apart respond differently to the same prompt in tone, length and tool-calling habits, and the final fortnight is a bad time to find that out.

## Incident response for quality regressions

A quality incident has no stack trace. Take a [support assistant](https://westack.ai/services/conversational-ai) that handles 3,000 conversations a day. One Tuesday the share escalated to a person rises by a third. There are no errors and latency is normal.

Ask what changed, in this order:

1. **Our release.** Compare the metric by release ID.
2. **The model.** Compare the model ID the provider reported before and after.
3. **The knowledge.** Check the index version. A re-index may have dropped or duplicated a key document.
4. **The traffic.** A product launch or an outage brings questions the assistant was never tested on.
5. **A dependency.** A tool's API changed its response, and the model is now reasoning from error messages.

Mitigate first: roll back the bundle, or route the affected topic to people. Then fix, add the failing conversations to the golden set and write it up.

In this case the traces show retrieval returning the previous returns policy. The nightly re-index had failed halfway and left an old index in service. With the index version in every trace, that takes an hour to find. Without it, days.

Have four things ready before you need them: a one-step rollback, a switch that hands a topic to people, someone on call who can read a trace, and a definition of severity for quality. A wrong refund amount and a slightly curt tone are not the same incident.

## Cost, latency and data retention

The dashboards we would not run without: cost per request and per outcome (per resolved conversation, say) by feature, customer and release; input, cached and output tokens; median and 95th-percentile latency by step; error and retry rates; and evaluation scores over time. Alert on the rate of change, not only on absolute levels. The budgeting side is in [FinOps for AI](https://westack.ai/blog/finops-for-ai).

Traces hold what users typed, which means personal data and sometimes payment or health details. Decide what content is captured, what is redacted on the way in, how long it is kept (short for raw text, longer for metadata and scores), who may read it, and how a deletion request reaches both the traces and any test set built from them.

Check the provider side as well. OpenAI, for example, [keeps abuse-monitoring logs for up to 30 days](https://developers.openai.com/api/docs/guides/your-data) by default and offers zero data retention only on approval. Terms differ by provider and change, so read the current page.

## How to start

In the first week:

1. Write the release manifest for what is running now, and pin the model ID.
2. Add tracing with the release ID on every request. Use OpenTelemetry-compatible instrumentation.
3. Collect fifty real cases into a golden set and run it by hand once. Wire it into CI the week after.
4. Put your models' retirement dates in a calendar with an owner.
5. Decide how long raw prompts and responses are kept.

For one application this is a week or two of an engineer's time. The standing cost afterwards is small and easy to skip: someone who knows the domain reading a sample of live conversations every week.

You will know it is working when three questions can each be answered in ten minutes. Which configuration produced this answer? Did quality change after the last release? What does one resolved conversation cost?

The first decision is who owns quality in production. If the answer is nobody in particular, no tool will help. Setting this up alongside the application is part of the [MLOps and LLMOps](https://westack.ai/services/mlops) work we do.

## Frequently asked questions

### What is LLMOps?

LLMOps is the set of practices for running applications built on large language models: versioning prompts, models, tools and retrieval settings together, tracing every request, evaluating quality before release and on live traffic, tracking cost and latency, and responding when quality drops. It borrows from DevOps and MLOps but deals with a model you usually do not control.

### How is LLMOps different from MLOps?

MLOps centres on training, deploying and retraining models you own, and on drift in your data. In LLMOps the model usually belongs to a provider who updates and retires it on their own schedule, behaviour is shaped by prompts, context and tools, and outputs vary between runs. The emphasis moves from training pipelines to evaluation, tracing and release control.

### Which tools do I need for LLMOps?

At a minimum: version control for prompts and configuration, a tracing backend that records each model and tool call, an evaluation runner wired into CI, and dashboards for cost, latency and quality. Prefer instrumentation based on OpenTelemetry so you can change backend later. The practices matter more than which product provides them.

### How do I know if my LLM app is getting worse?

Score a sample of live traffic against the same criteria you use before release, and watch user signals such as thumbs-down, retries, escalations to a person and abandoned sessions. Compare by release ID and by the model version the provider reports. A drop that lines up with neither your release nor theirs usually points to changed data or changed traffic.
