BlogGenerative AI

Agentic RAG and GraphRAG: when basic retrieval is not enough

Basic RAG runs one search and hopes the answer is in the top results. Agentic retrieval fixes questions that need several searches by letting the model plan, read and search again, while GraphRAG fixes questions about the corpus as a whole by building entities, relationships and community summaries ahead of time. Your failed queries tell you which one you need, and many teams need neither yet.

Where single-shot retrieval fails

Basic RAG does one thing. It turns the user’s question into a vector, finds the chunks of text nearest to it, and gives the top few to the model. For a question whose answer sits in one passage, phrased roughly the way the passage is phrased, this works well and should not be replaced.

Take a generic case: an engineering consultancy with 12,000 project documents, from tender responses to site reports. An engineer asks what concrete grade was specified on a named bridge project. One search, one passage, done.

Three kinds of question break it.

Multi-hop questions. “Which of our projects used the same piling subcontractor as the harbour job, and did any of them report delays?” No single passage answers that. The system has to find the subcontractor first, then the other projects, then the delay reports. A single search on the original wording retrieves a muddle of passages about piling and delays in general.

Whole-corpus questions. “What are the recurring causes of cost overruns across our projects?” A proper answer needs every project report. Vector search returns the five chunks that mention cost overruns most directly, and the model writes a confident summary of five documents out of 12,000. This failure is hard to spot because the answer reads well.

Vague questions. “Anything on the drainage issue?” Which project, which issue, which year? A person would ask a follow-up or try a few searches. Single-shot retrieval embeds the vague words and returns vaguely related text.

A caution before going further. Many apparent failures of basic RAG are failures of basic hygiene. If you have not yet added keyword search alongside vectors, a reranker, sensible chunking and metadata filters, do that first. Our RAG guide covers those steps, and they are cheaper than anything in this article.

What agentic retrieval adds

Agentic retrieval puts the model in charge of the search. Instead of receiving results, it is given search as a tool and decides how to use it. The loop looks like this:

  1. Read the question and plan: what needs to be found, and in what order?
  2. Issue one or more searches, often reworded from what the user typed.
  3. Read the results and judge them. Is this enough? Is something missing or contradictory?
  4. Search again with what has been learned, or stop and answer.

For the piling question, the model would first search for the harbour job’s subcontractor, read the name, search for that name across other projects, then search those projects for delays. Each step uses a fact the previous one produced, which a single search cannot do.

The same loop handles vague questions, because the model can start broad and narrow down. Anthropic’s write-up of its multi-agent research system gives this advice for its search agents: begin with short, broad queries, see what is available, then narrow the focus.

You do not always have to build the loop yourself. Managed search products now ship versions of it. Azure AI Search’s agentic retrieval, for example, can use a model to break a complex query into focused subqueries, run them in parallel, rerank each result set and merge the output. Its documentation is candid that this adds latency compared with a single-query pipeline. That simpler form, plan once and fan out, is a good halfway house: most of the benefit on multi-part questions, at a bounded cost.

A fuller agent with several tools (text search, a SQL query, a document fetcher, a date filter) goes further, and costs more to run and to test.

What a knowledge graph adds

Agentic retrieval still reads passages. It can chain facts, but it cannot read 12,000 reports to tell you the recurring themes. That needs work done ahead of time, across the whole corpus, and it is what GraphRAG is for.

The term comes from a 2024 Microsoft Research paper, From Local to Global: A Graph RAG Approach to Query-Focused Summarization. Its argument is that asking for the main themes of a dataset is a summarisation task over the whole corpus, not a retrieval task, so fetching a few passages is the wrong tool. The method has two indexing stages. First, a model reads the source documents and extracts entities and the relationships between them, building a graph. Second, the graph is grouped into communities of closely related entities, and the model writes a summary of each community in advance. At query time, each community summary produces a partial answer and the partial answers are combined.

The open-source GraphRAG project follows that shape: text units, extraction of entities, relationships and claims, hierarchical clustering with the Leiden algorithm, then community summaries built from the bottom up. It offers several query modes. Global search answers whole-corpus questions from the community summaries. Local search answers questions about a specific entity by fanning out to its neighbours. A mode called DRIFT adds community information to local search.

For the consultancy, the entities would be projects, clients, subcontractors, materials and types of incident. One community might gather marine projects, their piling subcontractors and tidal delays, with a summary already written. The cost-overrun question is answered from summaries that cover everything, instead of from five lucky chunks.

The graph helps multi-hop questions too, since “same subcontractor as the harbour job” is one hop along an edge. If your content has strong, nameable entities and you already hold some of them in structured form, a knowledge graph earns its place sooner.

What does each one cost?

Basic RAG Agentic retrieval GraphRAG
Indexing cost Embeddings only Same as basic A model reads every chunk, then writes summaries
Cost per query One search, one model call Several searches and model calls Low for local search, high for global
Latency Lowest Several model calls in sequence Global answers are the slowest
Multi-hop questions Poor Good Good where the graph has the edges
Whole-corpus questions No No Yes
When content changes Re-embed the changed chunks Same as basic Re-extract, and summaries go stale
Build complexity Low Medium: loop control, budgets, tracing High: extraction prompts, entity clean-up, storage

Two notes on what sits behind this table.

On agentic cost, Anthropic reports that in its own data, agents typically use about four times more tokens than chat interactions, and multi-agent systems about fifteen times more. Your ratio will differ. The direction will not. Each extra search round adds text that the model must read again on the next step, which is why context engineering matters so much in these loops.

On graph cost, full GraphRAG indexing is expensive because a model processes the entire corpus before anyone asks a question. Microsoft’s follow-up work, LazyGraphRAG, exists because of that cost. It skips the summarisation in advance and defers model use to query time, and Microsoft states that its indexing cost matches vector RAG and is 0.1% of full GraphRAG’s. If you are assessing graph approaches now, compare the lazy variants before committing to a full index.

How do you tell which one you need?

Your own failed queries are a better guide than any architecture diagram.

  1. Pull 100 real queries where users marked the answer as wrong, or asked again in different words.
  2. For each, check whether a passage containing the answer exists in your corpus. If it does not, no retrieval method will help. That is a content gap.
  3. If the passage exists and was not retrieved, and the question is a plain lookup, your basic retrieval needs work: hybrid search, reranking, chunking.
  4. If the answer needed facts from two or more places, where one fact leads to the next, mark it multi-hop.
  5. If the question was about the collection as a whole (themes, trends, comparisons across everything), mark it global.
  6. If the question was too vague to search for, mark it underspecified.

Then count. A pile of multi-hop and underspecified failures points to agentic retrieval. A pile of global failures points to a graph with community summaries. In the systems we work on, the first pile is usually much the larger, because most business users ask about specific things. Whole-corpus questions cluster in particular settings: research, compliance review, intelligence analysis, voice-of-customer work.

If both piles are small, you need neither yet.

What goes wrong in practice?

Agentic retrieval fails by not stopping. A model that is unsure keeps searching, and each round makes the answer slower and dearer. Set a hard budget on steps and tokens, and tell the model what to do when the budget runs out: answer with what it has and say what it could not find. Chat users will not wait through six rounds of searching, so send simple lookups straight to basic retrieval and keep the loop for questions that need it. You also need a trace of every step, because debugging means reading which searches were issued and why. Testing changes as well. You are now judging a sequence of decisions, which is the subject of our piece on evaluating AI agents.

GraphRAG fails at extraction. The model decides that “Acme Piling Ltd”, “Acme Piling” and “APL” are three entities, and your graph fragments. Default extraction prompts are written for general text and miss the entity types that matter in your field, so plan time to tune them on a sample. Updates are the other problem. New documents change the graph, and a community summary written last month no longer describes the community. For a corpus that changes daily, work out the re-indexing bill before you start.

Both add parts that someone has to own. A team that cannot yet measure the quality of its basic retrieval will not be able to tell whether either upgrade helped.

How to start

In the first week, do the failed-query exercise above and nothing else. It takes a few hours and it will change what you build.

If multi-hop questions dominate, add query planning with a fixed fan-out first: one model call that splits the question into subqueries, parallel searches, a reranker, one answer. Measure against the same 100 queries. Move to a full search loop only for what still fails.

If global questions dominate, pick 500 representative documents and index only those with a graph approach. Read the extracted entities yourself before you look at any answers. If the entities are wrong, the summaries built on them will be wrong as well.

Either way, keep basic retrieval as the default path and treat the new method as a route for the questions that need it. If you want help running that diagnosis, RAG and enterprise search is one of the things we build at WeStack AI.

Questions

Asked often.

What is agentic RAG?

Agentic RAG lets the model run the search instead of receiving one fixed set of results. It plans queries, reads what comes back, decides whether it has enough, and searches again if not. It handles questions with several parts or steps far better than a single search, at the price of more model calls and a slower answer.

What is the difference between GraphRAG and normal RAG?

Normal RAG retrieves text passages that resemble the question. GraphRAG first uses a model to extract entities and relationships from the whole corpus, groups them into communities and writes a summary of each. It can then answer questions about the corpus as a whole, which passage retrieval cannot do, but the index costs far more to build.

Is GraphRAG worth the cost?

Only if your users ask the questions it is built for: themes, patterns and connections across many documents. For lookups of specific facts, hybrid search with a reranker is cheaper and usually just as good. Check your failed queries first. If only a handful are whole-corpus questions, spend the effort on retrieval quality instead.

Do I need a graph database for GraphRAG?

Not necessarily. The published GraphRAG approach produces tables of entities, relationships, communities and summaries that can be stored as files or in an ordinary database. A graph database helps when you also want to query the relationships directly, or when the graph already exists for other reasons, such as a product catalogue or an organisation chart.

Not on the list? Ask us directly

Keep reading

More notes from the build.

All blogs

Let’s build your stack.

Send a brief and we’ll reply with next steps.