What is agentic RAG?
Agentic RAG is retrieval-augmented generation in which an AI agent controls the retrieval process instead of following a fixed pipeline. Classic RAG runs one search for every question and passes whatever comes back to the model. In agentic RAG the model reads the question, decides which sources to query and how, examines the results, and issues follow-up searches when the first ones fall short. It stops when it judges that it has enough evidence to answer, or reports that it could not find the answer.
The agent typically has several retrieval tools: a vector search over documents, a keyword search, a SQL query against a database, and sometimes a web search or a call to a business application through function calling. Choosing among them is part of the reasoning.
How agentic RAG works
The loop follows the ReAct pattern of reasoning and acting. The agent rewrites a vague question into precise sub-questions, routes each one to the right tool, and reads the results. If a passage references another document, it fetches that document. If two sources disagree, it looks for a third or flags the conflict. A final step composes the answer with citations to everything it used. Each tool call and intermediate decision is logged, which is what makes the behavior reviewable.
Agentic RAG in practice
A freight forwarder’s customer service team fields questions such as whether a shipment held at a port will clear in time for a delivery commitment. Answering requires the shipment status from the transport system, the customs hold reason from the broker’s portal, and the relevant clause in the customer’s service agreement. An agentic RAG system queries each source in turn, notices that the hold reason mentions a missing certificate, retrieves the forwarder’s procedure for that certificate, and drafts a reply with the expected clearance steps. A classic single-search RAG system would have returned only the service agreement.
What to watch for
- Multiple retrieval steps multiply latency and token cost, so set a cap on tool calls per question and measure the typical count.
- Every tool the agent can call needs the same permission checks as a human user of that system, applied to the user on whose behalf the agent acts.
- Insist on a trace of every search and result for each answer, because debugging a wrong answer is impossible without it.
- Start with classic RAG and move to agentic retrieval only for question types that measurably need several sources.