BlogAI agents

Long-running AI agents: checkpoints, compaction and recovery

An agent that works for hours fails in ways a thirty-second one never does. The process restarts mid-task, the context window fills, an approval takes until morning, and a retry sends the same email twice. The remedy is to treat each run as a durable workflow: save state after every step, make every action safe to repeat, keep progress in files the agent re-reads, and set budgets that stop it.

What breaks when a task takes hours

Take an accounts-payable agent that reconciles a month of supplier invoices. It has 4,000 invoices to match against purchase orders and delivery notes. It posts the clean ones to the ledger, flags mismatches, and asks someone in finance to approve anything above a threshold. That is about three hours of work and several thousand tool calls.

A chat request that fails can be sent again. This cannot, and the things that go wrong are different in kind:

  • The process dies. A deploy, an out-of-memory kill or a reclaimed instance ends it at invoice 2,317. Without saved state, the run starts again from the first invoice.
  • The context fills. Several thousand tool results do not fit in any window.
  • A person is slow. The approver is in a meeting, then at home.
  • Things happen twice. A retried step posts the same invoice again.
  • The agent drifts. After a few hundred steps it forgets a rule, or announces that it has finished when it has not.
  • Nobody can see it. Three hours of silence, then a result nobody can audit.

Each has a known fix, and most of the fixes come from workflow engineering, not from AI.

Durable state and checkpoints

The rule is that state lives outside the process and is saved after every step. A checkpoint holds the message history or a pointer to it, the step counter, the results of completed tool calls, any pending approvals, the spend so far and the list of items done.

You can build this as a table in your own database, with one row per step. Or you can use a durable execution engine, and several now target agents directly. They share one mechanism: record the result of every step, and after a failure run the code again from the top, skipping completed steps by handing back their stored results.

Temporal keeps an ordered event history for each workflow and replays it to rebuild state. AWS describes the same checkpoint-and-replay design for Lambda durable functions. Cloudflare Workflows persists the result of each step and retries failed ones. LangGraph’s checkpointers save a snapshot of the graph’s state at each step boundary.

Option How progress is saved Waiting Fits when
Temporal Event history, replayed after a failure Timers and signals; workflows can run for years You want a dedicated engine and will run or buy one
AWS Lambda durable functions Checkpoint and replay inside Lambda Waits suspend with no compute charge; runs last up to a year You already build on Lambda
Cloudflare Workflows Each step result persisted and retried Sleep for up to a year, or wait for an event You already build on Workers
LangGraph checkpointer State snapshot per step, in Postgres or SQLite An interrupt pauses the graph until resumed The agent is already a LangGraph graph
Your own table One row per step in your database A status column and a queue Linear agents and few dependencies

Replay has one consequence that surprises people: the orchestration code must be deterministic. Given the same history it has to make the same decisions, so anything that varies (the clock, random numbers, network calls and, above all, model calls) must happen inside a recorded step. The reward is that a model call recorded as a step is not made again on replay, which saves money and keeps the run consistent.

Check how often your framework actually writes. LangGraph documents three durability modes, and the lightest one saves only when the run ends. That is fine for a chat turn and wrong for a three-hour job. If you use an agent SDK, look for its own route: the OpenAI Agents SDK, for one, lists integrations with Temporal, Dapr, Restate and DBOS for runs that have to survive restarts and long waits.

Resuming after a crash

Loading the checkpoint is the easy part. The hard part is the step that was in flight. The process may have died after the ledger accepted invoice 2,317 and before the checkpoint recorded it. On restart, that step runs again. By default these engines retry a step whose result was never recorded, so plan for at-least-once execution, not exactly-once.

Pauses have the same property. LangGraph’s interrupt documentation warns that a resumed node restarts from its beginning, so any code before the pause point runs a second time.

Make every action safe to repeat

Assume each write can happen twice and design for it. We describe the principle in AI agents in production. For a long run it becomes specific:

  • Derive a key for every action from the run, the step and the item, such as run-88/post/INV-2317. Many APIs, payment providers especially, accept an idempotency key and return the original result when they see it again.
  • Read before you write where the API has no such key. Look up the invoice reference in the ledger before posting.
  • Prefer setting a state to adding one. “Set status to posted” is safe to repeat. “Add a posting” is not.
  • Send messages through an outbox. Write the email to a table with a unique key and let one sender deliver each row once.

Keeping the context useful over many steps

No context window holds 4,000 invoices’ worth of tool calls, and you would not want it to. Models become less reliable as their input grows, a problem we cover in context engineering. The agent has to work in batches, with the context cleared or compacted between them.

Compaction helps, and it loses detail every time. A summary of a summary of a summary will not remember that one supplier always rounds VAT down. So the durable record of the task should be files or tables the agent reads back, not the conversation.

Anthropic’s write-up on long-running agent harnesses is the most useful account of this we have read. Their agent built software across many context windows, and the pattern transfers directly:

  • A structured progress log. Theirs was a JSON list of features, each with a pass or fail field, and the agent was allowed to change only that field. Ours would be one row per invoice with a status and a reason.
  • A notes file for decisions and rules learned along the way.
  • A fixed start-of-session routine. Read the log, read the notes, check that the last batch really landed, then pick the next item.
  • One item at a time. They found the agent tried to do too much at once and, later in a project, declared the job finished on seeing how much was already done.

The last point matters most. “Done” has to be a check the harness runs: the ledger total matches, the test passes. The agent’s own opinion does not count. How the loop, compaction and hooks fit together is covered in designing an agent harness.

Waiting for people, events and schedules

An approval that takes fourteen hours should cost nothing while it waits. Do not hold a process open, and do not let the model poll in a loop. Persist the run, release the compute, and resume on an event. The engines provide this directly: Cloudflare Workflows has a wait-for-event step with a timeout, Lambda durable functions suspend during waits, Temporal uses signals and timers, and LangGraph has interrupts.

Three rules keep waits from causing trouble. Every wait has a timeout and a defined outcome when it expires: remind, escalate, or skip the item.

Do not block the whole run on one answer. Park the ten invoices that need approval, finish the other 3,990, and resume each parked one when its approval arrives.

And re-read the world after a wait. By the time the approver answers, the invoice may have been cancelled. An approval is permission to act on the state the approver saw, so check that the state still holds.

The same thinking applies to starting work. A long run is a background job: something puts it on a queue, a worker picks it up, and a cron schedule or an event triggers it. Take a lock per job so that two runs never reconcile the same month. Hosted agent platforms now offer the same pieces, with server-side sessions that resume, background runs you poll, and scheduled runs.

Budgets, timeouts and observability

A short agent needs a turn limit. A long one needs four budgets: steps, spend, wall-clock time, and attempts per item. Give every tool call its own timeout as well.

Make a budget stop a pause and not a crash. The state is saved, a person is told why, and they can raise the limit and resume. Add a stuck detector: no item completed in the last fifty steps, or the same error three times running.

Then make the run visible while it happens. Keep one trace per run, with a span for every model call, tool call and wait. Publish a progress figure that means something to the business, such as “2,310 of 4,000 reconciled, 14 awaiting approval, 3 failed”, beside the spend so far and the time of the last checkpoint. Raise an alert when a run that is not waiting has saved no checkpoint for fifteen minutes.

The finance lead should be able to open a page and see that progress line without asking an engineer. The tooling is the same as for any LLM system, which we cover in LLMOps in production.

How do you test a task that takes an hour?

Not by running it for an hour after every change. We use six tests.

  1. Shrink it. Thirty invoices that include every known kind of mismatch. It runs in minutes, on every change.
  2. Record and replay tool results. Saved responses make runs cheap and repeatable.
  3. Kill it on purpose. Stop the worker at random steps, including between an action and its checkpoint, restart it, and assert that the end state has no duplicates and no gaps.
  4. Force a context reset early. Set the compaction threshold very low so that it triggers at step ten, then check that the agent still knows the goal and the rules.
  5. Fake the clock. A 24-hour approval wait should take a millisecond in a test. Test the timeout path too.
  6. Run the full job nightly against a copy of last month’s data, compare the result with the known answer, and watch the trend in cost and duration.

The third test finds the most bugs, and it is the one we most often find missing.

Where to start

In the first week, put the run’s state in a table, checkpoint after every step, add idempotency keys to the two riskiest writes, and write the kill test. That alone removes most of the ways a long run loses work.

The first decision is your own table or an engine. Our rule: if the steps are linear and you already run Postgres, a table and a queue are enough. Once you need timers, signals, fan-out and waits that last days, adopt an engine, preferably the one native to the cloud platform you already use.

Whichever you choose, write down what “done” means as a check a program can run. It is the question we ask first when we scope an agent build at WeStack AI.

Questions

Asked often.

What is durable execution for AI agents?

Durable execution means the progress of a run is saved to storage after every step, so a crash, a deploy or a timeout does not lose the work. On restart the run continues from the last completed step. Workflow engines such as Temporal, AWS Lambda durable functions and Cloudflare Workflows provide it, as do agent frameworks with checkpointers.

How does an agent keep track of a task that outlasts its context window?

By writing its state outside the window. A structured progress log lists what is done and what remains, a notes file records decisions and dead ends, and compaction summarises older history. After a reset the agent reads those files first. The files, not the conversation, are the record of the task.

How do you stop an agent repeating an action after a crash?

Make every write idempotent. Give each action a key derived from the run and the step, pass it to APIs that accept idempotency keys, and for the rest record the action in your own database and check before acting. Recovery re-runs any step that was in flight, so assume each one can happen twice.

How long can an AI agent run?

There is no fixed ceiling. Workflow engines can keep a run alive for months, including time spent waiting for a person. The practical limits are budget, the quality of the agent's notes across context resets, and how well errors are caught. Set explicit limits on steps, spend and wall-clock time for every run.

Not on the list? Ask us directly

Keep reading

More notes from the build.

All blogs

Let’s build your stack.

Send a brief and we’ll reply with next steps.