Contents
In brief
One large-language-model call seems cheaper than a chain of five or ten. For support systems and internal knowledge bases, however, the useful number is not the cost of a single LLM call: it is the cost of resolving a question, including follow-up messages, waiting time, and human escalation.
The author describes a case in which an AI agent takes several independent retrieval steps yet costs less than a conventional RAG assistant. Retrieval-augmented generation, or RAG, supplies an LLM with passages retrieved from a knowledge base before it writes an answer.
What happened
A simple assistant usually follows a straight path: it converts the question into an embedding, retrieves relevant passages, and asks an LLM to compose a response. That is effective when the solution exists in one document. A difficult support case may instead require release notes, a migration guide, and an exact error message to be connected.
When the first response is incomplete, the user becomes part of the retrieval process. They clarify the question, reformulate the symptom, and wait for another answer. After several failed attempts, the case reaches an operator. Each model call might be inexpensive in isolation, but the question is still unresolved.
The project described in the article uses an agent that selects its next retrieval method. It can search semantically, look up an exact term, open a retrieved document passage, and then produce an answer with citations. This uses more LLM calls, but it avoids making the user steer the search through repeated follow-ups.
Why it matters
The proposed measure is the complete cost of a resolved question. It includes tokens, the user’s time, the rate of escalation to an operator, and the operator’s time. Extra model work can be economical when it reduces the number of questions that require a person.
The article illustrates this with an error after a product update. A conventional assistant repeatedly suggests checking general mail settings, then finds a renamed field only after the user supplies more detail. The agent notices a likely connection to release notes, opens them, searches for the new field name, and finds the migration instructions.
This does not make multi-step retrieval the right choice for every request. A direct factual question, such as which port a service uses, is cheaper and faster with a simple answer. Agents are most useful when an answer must be assembled from multiple sources or competing explanations must be checked.
In practice
The tool set is only part of the design. An agent needs limits so it can stop an unproductive search, preserve the original question, and avoid filling its context with every intermediate result. In the reported system, continuously summarizing history with an LLM was expensive and sometimes lost details that mattered.
The alternative was a sliding context window. The system instruction and latest messages remain, while older tool results are removed once the context reaches a limit. Summarization remains a restricted fallback. It is less elegant, but more predictable for a production support system.
- Keep exact-text and semantic search separate: error codes and field names need string matching, while broad issues benefit from semantic retrieval.
- Use different model tiers for lightweight steps, final synthesis, and embedding generation.
- Record calls, tokens, tool results, time to answer, and operator escalations for every session.
- Evaluate simple, complex, and ambiguous questions independently; one mode will not be best for all three.
- Measure first-answer resolution rate, not only the number of calls made by the model.
Takeaway
An AI agent does not replace simple RAG assistants. For short reference questions, extra reasoning increases both latency and token use. It also cannot compensate for an empty or outdated knowledge base: repeated retrieval cannot discover information that was never stored.
For support cases with compound symptoms, the cost of one answer is the wrong metric. A system that finds the cause, the repair instructions, and the supporting documents by itself can save user and operator time as well as tokens. Compare these systems by the cost of a resolved question and the share of cases actually resolved.