Contents
In brief
An LLM pilot costs cents. Then agents, retries, and “add a model to every feature” grow the bill faster than revenue. A Dev.to piece walks through the levers that usually cut 30–60% of runtime spend without hurting answer quality: semantic caching plus provider prompt caching, routing easy work to small models, and token discipline. Cache first, route second, trim third — only then consider a weaker model.
What happened
Picture a kitchen that remakes the same dish from scratch every time a guest rephrases the order. LLM apps do that with meaning: the same intent hits the API again and again, while a flagship model prices greetings and labels like hard reasoning.
The Dev.to write-up treats cost as a feeding problem, not a shopping problem for the cheapest model. A semantic cache stores recent answers keyed by meaning, so a rephrased question can hit the cache instead of the model. Separately, provider prompt caching bills a shared system-prefix once when the same “header” repeats call after call.
Routing is the structural lever. Classification, short summaries, tags, and simple extraction run well on a small model; keep the flagship for codegen, edge cases, and multi-step agent planning. This is often the largest structural win: you change how many expensive tokens you buy, not only the price per token. On objective batches (labels, extraction) where quality is easy to check, routing pays especially well.
Token discipline closes the silent leaks. Send relevant chunks instead of the whole document. Cap output length: output tokens are often the expensive side. Trim system rules that do not earn their weight. A shorter, higher-signal context plus a controlled reply is the cheapest fix — and it needs no new tool.
Measurement is not a saving by itself, but without it the rest swings at the wrong target. Log model, tokens, and cost per feature or team. Otherwise you “optimize everything” while a background job hammers a flagship per object, or a chatbot resends ten thousand context tokens every turn.
Why it matters
The invoice is rarely the table price per million tokens. It is what you send and how often you repeat the same work. Teams often downgrade the model first and accept quality risk, even though the cheap levers sit on the surface and hold longer than a weaker model.
Diagnosis beats the toolkit. If the bill is dominated by the same call thousands of times, that is a caching problem. If a few heavy agent runs dominate, that is routing and model choice. Caching flashcard-style questions will not fix a sync job that runs an expensive analysis on every object.
The strongest cost project is a loop, not one lever. A month of logs, a budget flag on the top three features, then caching, routing, and trimming aimed at the spikes the numbers show. After a few cycles the bill stops being a mystery and becomes a line item with a reason.
In practice
Same kitchen order: do not remake what you already have, do not call the chef for a salad, do not haul the whole pantry to the table — then argue about hiring a different chef.
- For a month, log every call: model, tokens, cost, tied to a feature or team. Find the top three lines.
- For chat and frequent-question flows, add a semantic cache by meaning and enable provider prompt caching when the system prompt is long and stable.
- Put an intent router in front: labels, greetings, and short summaries on a small model; code, hard reasoning, and agent planning on the flagship.
- Plug leaks: retrieve relevant chunks instead of dumping the document, hard-cap output length, keep a short system prompt without dead rules.
- Only after numbers and the three levers above compare a weaker model — and measure cost per closed task, not token list price alone.
| Task | Small model | Flagship |
|---|---|---|
| Greetings, labels, tags | Yes | Rarely |
| Summarize a short text | Yes | Sometimes |
| Codegen, hard reasoning, edge cases | No | Yes |
| Agent planning, long instructions | No | Yes |
Takeaway
A cheap token does not save a bloated context or a thousand duplicate answers. The bill moves when you cache duplicates, route the trivial to the cheap model, and shorten what you ship — in that order. Once every invoice line has a number behind it, a weaker model is a conscious trade-off, not a panic move on list price.
Comments