A demo shows that a model can do a task once, on inputs you chose, while you watch. Production means it does the task thousands of times on inputs you did not choose, recovers when a tool fails halfway through, stays inside a budget and leaves a record someone can audit. In our experience the demo is about a fifth of the work, and the rest is the engineering described here.
Why the demo is only a fifth of the job
The demo is the easy part because current models are good. Give one a clear instruction and three tools and it will complete a sensible task on the first afternoon. That is real progress, and it is also why so many teams misjudge what remains.
Take a scenario we will use throughout. A distributor’s accounts-payable team answers supplier emails asking where a payment is. Someone reads the email, finds the invoice in the ERP, checks it against the purchase order and the goods receipt, and replies with a payment date or an explanation of what is blocking it. An agent that does this on ten sample emails takes a couple of days to build.
Then the real inbox arrives. One email asks about four invoices and attaches a statement as a scanned PDF. Another quotes an invoice number with two digits swapped. The ERP times out on the third lookup. A supplier replies to the agent’s answer with new bank details, which is either routine or fraud. Volume is 400 emails a day, so a failure that appears once in a hundred runs happens four times before the day is out.
None of that is a model problem. It is the work any production system needs, plus a few things specific to software that decides its own next step. The rest of this article goes through that work in order.
Scope the agent to one job, then design its tools
One job with a finish line
The agents that reach production have a job description you could hand to a new hire. Ours would say: you answer supplier payment-status queries. You may read invoices, purchase orders, receipts and payment runs. You may draft a reply. You may not change bank details, approve invoices or promise a payment date that is not in the system. If the email is about anything else, pass it to the AP queue with a one-line summary.
That paragraph does more for reliability than any prompt technique. It defines when a run is finished, which makes it testable. It limits the tools, which limits the damage. And it gives the agent somewhere to send everything it should not handle, which matters because declining well is half of the job.
Be honest about how much of the task needs an agent at all. Anthropic’s guide to building effective agents separates workflows, where code fixes the sequence of steps, from agents, where the model chooses its own steps, and recommends ‘finding the simplest solution possible, and only increasing complexity when needed’. We agree. In the AP example, classifying the email and extracting invoice numbers is a fixed pipeline. Only the investigation in the middle, where the next lookup depends on what the last one returned, needs the model to steer.
Tools written for the model, not copied from your API
Most demo agents are given thin wrappers around existing endpoints: get_invoice, get_po, get_goods_receipt, list_payment_runs. The model then has to work out the join logic that your finance team took years to learn, and it will get it wrong in the cases that matter.
A better tool matches the task. get_invoice_status(invoice_number) returns one object with the invoice, its matching state, what is blocking it in plain words, and the scheduled payment date if there is one. Anthropic reached the same conclusion from its own testing. Its write-up on writing tools for agents warns against wrapping every endpoint, on the grounds that ‘more tools don’t always lead to better outcomes’, and asks for error messages that tell the agent what to fix.
Our working rules for tools:
- Few tools, each named for an outcome a person would recognise.
- Read tools and write tools are separate, so their permissions can differ.
- Arguments are validated in code. The supplier ID comes from the authenticated sender, never from the model’s guess.
- Results are short and use names, dates and amounts, not internal keys.
- Errors say what happened and what to try: ‘No invoice 40187 for this supplier. Closest matches: 40178, 41087.’
State, retries and long-running work
Keep state outside the model
A chat demo keeps everything in the context window. A production run cannot. The process may be restarted mid-run, a human approval may take six hours, and a long investigation can outgrow the context.
So each run gets a durable record: the trigger, the steps taken, every tool call with its arguments and result, and the current status. The record is written after each step and before the next one starts. If the worker dies, another picks the run up from the last completed step instead of starting again and repeating actions. Whether you use a workflow engine built for durable execution or a queue and a database table matters less than doing it from the first version. Retrofitting state into an agent that assumed it would always finish inside one request is a rewrite.
Make every write safe to repeat
Tool calls fail in an awkward way: the request times out and you do not know whether the action happened. The agent, or your retry wrapper, will try again. If the tool was ‘send reply to supplier’, the supplier now has two emails. If it was ‘release payment’, you have a worse problem.
The fix is old and well understood. Every write carries an idempotency key, and the system receiving it refuses to perform the same keyed action twice. Stripe’s API documentation describes idempotent requests as the way to retry safely without performing the same operation twice. For an agent, derive the key from the run and the step, never from anything the model generates:
{
"tool": "send_supplier_reply",
"idempotency_key": "run_8f3a:step_6",
"arguments": { "thread_id": "T-20931", "draft_id": "D-5512" }
}
Then sort errors into two kinds. Timeouts, rate limits and server errors are retried by your code with backoff, a small fixed number of times, without involving the model. Validation errors go back to the model with a useful message, because it may be able to correct the call. Anything still failing lands in a queue that a person looks at. An agent that gives up silently is worse than one that never ran.
Where should a human approve?
Approval is not a general property of the system. It attaches to particular actions. We sort every tool into three groups:
- Reads: no approval, logged.
- Reversible internal writes, such as a draft, a note on the invoice record or a ticket: no approval once shadow results are good, logged and sampled.
- Irreversible or external actions, such as an email to a supplier, a released payment hold or a change to master data: approval required until the evidence says otherwise, and for some actions permanently.
In the AP example, changing bank details is never an agent action. The agent’s only move is to flag the request and route it to the people who run the call-back check.
The approval screen deserves design effort. The reviewer should see the exact action and its arguments in plain language, the evidence the agent used, and three choices: approve, edit, or reject with a reason. Edits and rejections are the most valuable data the system produces, since each one is a labelled failure you can turn into a test case.
Watch for approval fatigue. If someone clicks approve two hundred times a day without reading, you have a ritual and no control. Either the action is safe enough to automate for a defined category, or the screen is not showing what the reviewer needs. We cover the security side of this, including prompt injection, in our article on guardrails for AI agents.
Evals, tracing and cost ceilings
These three let you change the system without fear. Without them every prompt edit is a gamble.
Evals before launch
Before any user sees the agent, build a task set from real cases: actual supplier emails with the outcome a competent clerk reached. Grade the end state (was the right status reported, was the fraud attempt escalated), run each case several times because results vary between runs, and make the suite a gate for every change to prompts, tools or model version. The full method is in how to evaluate AI agents.
Tracing every run
When a supplier complains about an answer three weeks later, you need to open that run and see each model call, each tool call with arguments and results, token counts, timings, and which prompt and model version were live. Log these as structured traces from day one. OpenTelemetry maintains semantic conventions for generative AI covering model calls and MCP, which saves you from inventing your own field names and lets agent traces sit beside the rest of your application telemetry. The conventions are still changing, so pin the version you emit. Tracing is the first piece of LLMOps to put in place, and the hardest to add later.
Cost ceilings
An agent in a loop spends money until something stops it. OWASP’s AI agent security cheat sheet lists this as denial of wallet from unbounded loops, and it does not need an attacker. A tool that returns an unhelpful error is enough.
Put the limits in code. A budget of steps, tokens and minutes per run, with a rule for what happens when it is hit: stop, save, and hand the run to a person with the trace attached. A daily ceiling per agent. An alert when any run is cut off, and a single switch that pauses the agent. Then track cost per completed task, not cost per call, because a cheap model that needs three attempts is not cheap. The wider budgeting picture is in what an AI agent really costs.
Rollout: shadow, assisted, then autonomous
Do not go from demo to autonomous. Each stage produces the evidence that justifies the next.
| Stage | What the agent does | What you measure | When to move on |
|---|---|---|---|
| Shadow | Runs on live inputs and changes nothing. Proposed actions are logged beside what staff did. | Agreement with staff decisions, types of disagreement, cost and time per run. | Disagreements are understood, and the serious ones are fixed and added to the eval suite. |
| Assisted | Prepares the action. A person approves, edits or rejects it. | Share approved without edits, reasons for rejection, time saved per case. | A defined category of cases is approved unchanged, consistently, for several weeks. |
| Autonomous, by category | Acts alone on the proven categories. Everything else stays assisted. | Sampled audit results, escalation rate, reversals and complaints. | Never fully. Sampling and review continue. |
Two notes from doing this. Shadow mode often shows that staff disagree with each other, and that the written policy is not what people do. Settle that before blaming the agent. And autonomy is granted per category, not per agent: ‘invoice paid, here is the date’ can go unattended long before ‘invoice blocked on a price mismatch’ does.
Production-readiness checklist
- The agent has a written job description with a finish condition and an out-of-scope route.
- Each tool is named for an outcome, validates its arguments in code and returns actionable errors.
- Read and write tools are separate, and the agent’s credentials allow only what the job needs.
- Run state is persisted after every step, and a run can resume after a restart.
- Every write takes an idempotency key that the model does not generate.
- Retries are bounded, and failed runs land in a queue that a person owns.
- Each action is classed as read, reversible or irreversible, with an approval rule for each class.
- An eval suite built from real cases runs on every change and blocks release when it regresses.
- Every run produces a trace with model calls, tool calls, cost, latency and version labels.
- Per-run and per-day budgets are enforced in code, with alerts and a pause switch.
- The agent has run in shadow on live inputs and the disagreements have been reviewed.
- A named owner reads the failures every week.
If you cannot tick items 4 to 6, you have a demo with a nicer interface.
Where to start this week
Pick one job with a clear finish line and enough volume to matter. Export fifty real cases along with what your staff did in each. Write the one-paragraph job description and ask the people who do the work to correct it. They will.
Then build the read-only version first: the tools that look things up, the run record, the trace. Put it in shadow against the live queue and read the disagreements every morning for a week. By the end of that week you will know whether the agent is close, what the tools are missing, and which actions will need approval. That is a far better basis for a go or no-go decision than a demo, and it is how we begin the AI agent builds we take on at WeStack AI.