BlogGenerative AI

Context engineering: what it is and how to do it well

Context engineering is the work of choosing, ordering and trimming everything a model reads before it answers: instructions, tools, documents, examples, history and memory. A model can only use what is in its context window, and it gets worse when that window is padded with things it does not need. Doing it well means giving each call the smallest set of information that is enough, and checking by reading what was actually sent.

What is context engineering?

A language model has no memory of your business and no view of your systems. On each call it sees one thing: the block of text, and sometimes images, that your software sends. That block is the context window. Everything the model appears to know about the user, the task and the conversation so far is in there because your code put it there.

Context engineering is the work of deciding what goes in. Anthropic, whose guidance on the subject is the fullest we have read from a model provider, describes it as curating and maintaining the set of tokens a model sees at inference time, as opposed to writing one good prompt. Put more plainly: for this call, what should the model be looking at, in what order, and what should be left out.

It grew out of prompt engineering, and the difference is scope. Prompt engineering is about wording instructions. That still matters, and we cover it in our prompt engineering guide. But in a real product the instructions are a small share of what the model reads. A support assistant’s request might be one tenth instructions and nine tenths tool definitions, retrieved articles and chat history, all assembled by code at run time. An agent repeats that assembly at every step, and the quality of the assembly decides the quality of the answer.

It needs care because more context is not better. Anthropic’s article treats context as a finite resource with diminishing returns: as the window fills, the model’s ability to recall any one item in it falls. Its API documentation calls the effect context rot. A study by the vector database company Chroma, which tested 18 models, found that performance became less reliable as inputs grew, even on simple tasks, and that text which resembles the answer but is wrong does particular damage.

What goes into a context window?

Six kinds of content compete for the space.

Component What it is for How it goes wrong
System instructions Role, rules, tone, output format Grows by accretion, rules contradict each other
Tool definitions What the model can do and how to call it Too many tools, overlapping purposes, verbose schemas
Retrieved documents Facts the model does not have Too many chunks, near-duplicates, stale versions
Examples Showing the behaviour you want A long list of edge cases that the model copies too closely
Conversation history Continuity across turns Sent in full for ever, old tool output included
Memory and notes Facts that outlive one conversation Everything saved, nothing pruned, wrong facts persist

System instructions

Write for a capable new colleague. Anthropic’s advice is to aim between two failures: brittle if-this-then-that logic on one side, vague high-level guidance on the other. Start short, test, and add an instruction only in response to a failure you have seen. Group the prompt into labelled sections so that the model and the next engineer can both find things.

Tool definitions

Tools are the part teams most often forget to count. The name, description and parameter schema of every tool are sent on every call. Anthropic’s tool search documentation gives the example of a five-server setup that consumes around 55,000 tokens in definitions before any work is done, and says the model’s ability to pick the right tool degrades beyond 30 to 50 tools. The fixes are to merge overlapping tools, to load definitions only when they are needed, and to make tool results short by default. If a person reading the tool list could not say which one to use, the model will not manage it either.

Retrieved documents

A RAG pipeline is context engineering done by a search system. The usual fault is quantity: ten chunks when the answer is in two. Rerank and cut. Remove near-duplicates. Label each passage with its source and date so that the model can prefer the newer one. For harder questions, let the model fetch what it needs step by step instead of loading everything up front, which is the idea behind agentic retrieval.

Examples, history and memory

A few varied examples teach better than a page of rules. Three to five that differ from each other are worth more than twenty that look alike.

History needs managing. Keep recent turns in full, summarise older ones, and drop bulky tool results once the model has used them.

Memory is anything written outside the window and read back later: a user’s preferences, or a file of notes an agent keeps during a long task. It needs the same pruning as everything else, and a way to correct an entry that turns out to be wrong.

Why does the order matter?

Position affects both quality and price.

For quality, when the context is long, put the question after the material. Google’s long-context guidance for Gemini says performance is better with the query at the end of the prompt, after all the other context.

For price, providers cache the start of a prompt and bill a repeated prefix at a fraction of the normal rate, but only when the prefix is identical. So the stable parts (tools, instructions, shared reference material) go first, and the parts that change (the passages retrieved for this question, the new message) go last. One timestamp at the top of the system prompt undoes it. We explain the mechanics in cutting LLM costs.

The two rules agree: fixed material first, the specific question last.

How do you keep long tasks on track?

A chat that lasts five turns can carry its whole history. An agent that works for an hour and makes a hundred tool calls cannot. Four techniques handle that, often together.

Clearing old tool results. Much of an agent’s context is tool output it has already acted on: file contents, search results, API responses. Once used, these can be replaced with a short placeholder. Anthropic offers this as a server-side feature called context editing, and it is easy to do in your own code.

Compaction. When the history nears the limit, summarise it and carry on from the summary. Everything depends on what the summary keeps: decisions made, problems still open, facts established. Anthropic and OpenAI both provide compaction on the server. One trade-off in OpenAI’s version is worth knowing. Its compaction guide describes the compacted item as opaque and not meant to be read by people, which makes a bad hand-over harder to diagnose. If you need to inspect what was kept, write your own summarising step.

Notes outside the window. Have the agent write its progress to a file or a store, such as a to-do list or the findings so far, and read it back when needed. Notes survive compaction and restarts.

Sub-agents. Give a self-contained piece of work to a separate model call with a clean context, and take back only its short result.

Each has a cost. Clearing and compaction can discard something that turns out to matter later, and both change the prompt prefix, which invalidates the cache from that point on. Sub-agents multiply token use. Add them once a task is failing for lack of room.

A worked example: trimming a bloated context

Take a generic case of a pattern we meet often. An online retailer has an order-support assistant. It works, but answers are slow, the bill is high, and it sometimes quotes a returns policy that was replaced months ago. We log one full request and count the tokens in each part. The figures are illustrative.

Component Before After What changed
System prompt 4,200 1,400 Removed rules for retired promotions and three conflicting tone instructions
Tool definitions 9,500 (38 tools) 2,100 (7 tools) Merged the order tools, dropped admin tools the assistant never needs
Retrieved policy text 8,000 (10 chunks) 2,400 (3 chunks) Added a reranker, filtered out superseded versions by date
Examples 3,000 (15 examples) 900 (4 examples) Kept four that differ from each other
Conversation history 6,300 2,000 Summarised turns older than the last four, dropped raw order JSON
User message 60 60 Unchanged
Total 31,060 8,860

Three things happened. The request shrank to under a third of its original size, so it is cheaper and faster. The outdated policy stopped appearing, because the superseded document is no longer retrieved. No instruction telling the model to prefer the new policy would have been as reliable as removing the old one. And tool selection improved, because seven distinct tools are easier to choose between than 38.

Nothing here needed a different model. Each change came from reading the request and asking, of every block, whether the model needed it for this question.

One caution: trim against a test set. Remove a block, rerun 50 real conversations, compare. Some of what looks like padding turns out to be holding a behaviour in place.

How to debug: read what was actually sent

When a model gives a bad answer, look first at the exact request: the rendered text with every variable filled in. The prompt template and the code that builds it will not show you the problem.

  1. Log the complete request and response for every call, with token counts per component.
  2. For a failing case, open the request and read it from top to bottom as if you were the model.
  3. Ask whether the information needed to answer correctly is present. If it is not, you have a retrieval or tooling problem.
  4. If it is present, ask what else is there that points the wrong way: a contradicting rule, an old document, an example that fits badly.
  5. Check the order. Is the question buried in the middle? Is the relevant passage the ninth of ten?
  6. Change one thing, rerun the test set, and keep the change only if the numbers move.

Most of the surprises come at step 2. Templates hide a great deal: a history that was meant to be capped and is not, a tool list that includes every tool in the registry, a retrieved chunk that turns out to be 4,000 tokens of navigation menu. An observability tool that shows the full trace of each call pays for itself here.

How to start

In the first week, pick your most used or most expensive feature and capture twenty real requests in full. Build the same table as the worked example: tokens per component. Most teams have never seen this breakdown for their own product, and it usually makes the first three fixes obvious.

Then set a budget per component and enforce it in code: a cap on retrieved tokens, a cap on history, a fixed list of tools per feature.

Make the first decision an organisational one: who owns the context for each feature. Without an owner, every team appends and nobody removes. If you want a second opinion on an assistant or agent that has grown unwieldy, that review is part of the conversational AI and agent work we do at WeStack AI.

Questions

Asked often.

What is context engineering?

Context engineering is the practice of deciding what goes into a model's context window on each call: instructions, tool definitions, retrieved documents, examples, conversation history and saved notes. The aim is the smallest set of information that lets the model do the task well, assembled fresh for every step.

How is context engineering different from prompt engineering?

Prompt engineering is mostly about wording the instructions. Context engineering covers everything else the model reads as well, and treats it as something a system assembles at run time. In an agent that makes fifty calls in one task, the instructions are a small part of each call. What surrounds them decides the outcome.

Does a bigger context window make context engineering unnecessary?

No. A larger window raises the limit on what you can send, but models still become less reliable at recalling details as the input grows, and every token is paid for on every call. Bigger windows make it easier to be careless. The discipline of sending only what the task needs still pays off.

What is compaction in an AI agent?

Compaction replaces a long conversation history with a shorter summary so that a task can continue past the context limit. Good compaction keeps decisions, open problems and established facts, and drops raw tool output that has already been used. Anthropic and OpenAI both offer it as a server-side feature, and you can also write your own.

Not on the list? Ask us directly

Keep reading

More notes from the build.

All blogs

Let’s build your stack.

Send a brief and we’ll reply with next steps.