What is agentic RAG?

Agentic RAG is retrieval-augmented generation where an LLM agent controls the retrieval step. Instead of always retrieving once and then generating, the agent decides whether to retrieve, which source or tool to use, whether the results are good enough, and whether to search again. A 2025 survey defines it as embedding autonomous AI agents into the RAG pipeline.

For background, see what are AI agents and what is an agentic workflow.

From RAG to agentic RAG

The original RAG paper (Lewis and colleagues, submitted May 2020) paired a pre-trained seq2seq generator with a neural retriever over a dense vector index of Wikipedia. Most RAG systems since follow the same shape: retrieve, then generate.

The agentic RAG survey says standard RAG relies on static workflows that struggle with multi-step reasoning. Its authors name four design patterns agents bring to the pipeline: reflection, planning, tool use and multi-agent collaboration (arXiv, v4 revised April 2026).

	Classic RAG	Agentic RAG
Retrieval	Fixed step, usually one pass	The model decides whether and when to retrieve
Query	The user’s question	Can be rewritten or split by the model
Quality check	None by default	Retrieved documents can be graded, with a retry on a miss
Sources	Usually one index	The retriever is one of the tools the agent can call
Cost and latency	Lowest	Higher, since each decision is a model call

The last row follows Anthropic’s general point that agentic systems often trade latency and cost for better task performance. Azure’s docs say the same of their own pipeline: agentic retrieval adds latency compared to a single-query pipeline, but handles query complexity a single query cannot.

What the loop looks like

The LangGraph agentic RAG tutorial is a concrete reference. It builds “a custom retrieval agent” that “decides when to search a vector store versus answering the user directly.” Its graph looks like this:

 question
    |
    v
 generate_query_or_respond
    |        \
    | tool    \ no tool call
    v          v
 retrieve     answer directly
    |
    v
 grade_documents
    |        \
    |relevant \ not relevant
    v          v
 generate_    rewrite_question
 answer         |
                +--> back to the top

The model answers directly or calls a retriever tool. A grader labels the retrieved text relevant or not. If it is not, the question is reworded and the loop runs again. The tutorial says “This retry loop is how the agent recovers from a weak first retrieval.” The structure is a ReAct-style loop with retrieval as the tool.

Research methods behind the pattern

Two papers define the decisions an agentic RAG system adds.

• Self-RAG (Asai and colleagues, October 2023) trains one language model to decide when to retrieve, then generate and critique both retrieved passages and its own output using reflection tokens. The authors report their 7B and 13B models beat ChatGPT and retrieval-augmented Llama2-chat on open-domain QA, reasoning and fact verification.
• Corrective RAG (Yan and colleagues, January 2024) uses a lightweight retrieval evaluator that scores the retrieved documents and produces a confidence level, which triggers the next retrieval action. When a static corpus returns weak results, it supplements them with web search, and it filters irrelevant content from the documents before generation. The authors call it plug-and-play with other RAG methods.

Self-RAG changes the model through training. Corrective RAG wraps an existing pipeline. Most framework tutorials, including LangGraph’s, take the second route and use prompts and graph edges.

A managed example: Azure AI Search

Retrieval services now ship the pattern. Microsoft’s agentic retrieval in Azure AI Search (page dated September 2026) works like this:

1. Your app sends a query and conversation history to a knowledge base.
2. An LLM plans focused subqueries, using chat history for context. This planning step is marked preview, and at minimal reasoning effort it is skipped.
3. The subqueries run in parallel, and each is semantically reranked.
4. The results are merged and returned, with optional source references and an activity log.

Microsoft says this suits multi-part questions, questions that depend on earlier turns, and queries that benefit from rewriting. Billing is token based, from Azure AI Search for execution and from Azure OpenAI for planning. The activity log shows which queries ran against which sources, which helps when you estimate cost.

Here the agent loop sits inside the search service instead of your code. Check the page for current preview and GA status before you rely on a feature, since preview features carry no SLA.

When to use it

Anthropic recommends finding “the simplest solution possible” and adding complexity only when needed. It also notes that for many applications, optimizing single LLM calls with retrieval and in-context examples is usually enough. A reasonable order, following that advice:

1. Start with single-pass RAG and measure where answers fail.
2. If failures come from weak retrieval on one index, add a grader and a query rewrite step.
3. If questions span several sources or need several lookups, let an agent choose among retrievers.

Anthropic draws the same line: a fixed workflow fits when the steps are known, and an agent fits when the model must choose the steps. How to build AI agents covers that decision.

Open problems

The survey lists unresolved areas: evaluation, coordination, memory management, efficiency and governance (arXiv). In practice each extra decision is another place to fail, such as a wrong grade, a poor rewrite or a loop that never ends. Cap the number of retries, as Anthropic advises with maximum-iteration stopping conditions for agents.

Tracing an agentic RAG run

Each retrieval, grade, rewrite and generation is a separate step, so a trace is the way to see why an answer was wrong. See agent observability.

HoneyHive’s docs cover tracing LangGraph state graphs, node transitions and LLM calls, and LlamaIndex query engines, retrieval, embeddings and LLM calls. For logic the automatic instrumentation misses, custom spans use the @trace decorator or context managers.

FAQ

What is the difference between RAG and agentic RAG?

Classic RAG retrieves once and then generates. Agentic RAG puts an agent in control of retrieval, so it can pick tools, rewrite queries, check results and retry. The 2025 survey on agentic RAG describes it as embedding autonomous agents into the RAG pipeline to overcome the static workflows of standard RAG.

Is agentic RAG always better than RAG?

No. Anthropic says that for many applications, optimizing single LLM calls with retrieval and in-context examples is usually enough, and that agentic systems often trade latency and cost for better task performance.

What are Self-RAG and Corrective RAG?

Two research methods that add decisions to retrieval. Self-RAG trains a model to decide when to retrieve and to critique passages and its own output. Corrective RAG uses a retrieval evaluator and falls back to web search when retrieval is weak.

How do you build agentic RAG?

A common pattern is a graph where the model either answers or calls a retriever tool, a grader checks the documents, and a rewrite step retries on a miss. The LangGraph agentic RAG tutorial builds exactly that.