← All posts

Cheap LLMs for agents: why token savings raise task cost

Why distilled models spawn extra tool calls, how benchmarks mislead, and which metrics beat price-per-million for agent TCO.

Tags
agentsAIarchitectureNewsllm
Cheap LLMs for agents: why token savings raise task cost
Contents

In brief

A Habr piece (Bothub) unpacks an agent paradox: models get cheaper while tasks finish worse. Lightweight variants sell on price per token, then burn more tool calls, loop, and inflate the bill. The fix is to measure cost per completed task, tool-call efficiency, and failure rate—not list price per million tokens.

What happened

Providers shrink models (distillation, quantization, pruning) to serve exploding demand on the same silicon. On short exams like MMLU the “almost flagship” gap looks fine. Agents need planning and multi-step recovery—exactly what gets cut first.

A familiar scenario: find a file, extract data, post a report. A strong model finishes in a few steps; a cheap one searches, debugs, and rewrites until tokens and wall time erase the “savings.” The author also attacks benchmarks: training contamination, saturated old suites, and weak link to enterprise pipelines (EnterpriseOps-Gym tops finishing only about a third to half of realistic flows).

Prefix-cache miss compounds the tax: each tool reply nudges context, cache hits drop, and the full prompt is re-read.

Why it matters

Teams pick models from a “cheaper and almost as good” table. In an agent product that becomes an expensive script pretending to work. Gartner already forecasts mass cancellations of agent projects from cost blow-ups and weak risk control—exactly where nobody computed agent TCO.

In practice

  1. Build a small set of your production scenarios and run candidates repeatedly (agents are non-deterministic).
  2. Track cost per task with retries, tool-call efficiency, and failure rate—not only $/1M tokens.
  3. Log every agent step; without traces you cannot see where the cheap model thrashs.
  4. Distrust single runs and single leaderboards; look at medians and spread.
  5. In multi-agent chains remember error multiplication; “98% per agent” is not system reliability.

Takeaway

A cheap token is not a cheap agent. Until you meter the path to a closed task, model “optimization” stays marketing. The article is a solid metrics frame before the next production downgrade.