What happened

The team walked five configs—from a dense 30B to an 80B MoE—then added token breakdown via LiteLLM, and only then enabled cache for their hybrid architecture. Before the gateway they had one “tokens spent” number that explained nothing. With breakdowns, input/output ≈ 452:1 appeared: a typical agent step carries ~124k input tokens and emits ~274, with almost all input served from cache.

Economics are candid: rent 131k ₽/month; on raw tokens the cloud often wins; on seats and subscription windows the story flips. The unintended conclusion: self-hosted inference is bought for perimeter and availability, not token savings. The stand’s ceiling is KV pool size (8–9 concurrent full contexts), not tok/s.

The piece also lists Blackwell flags, three footguns (DeepGEMM on sm120, required prefix-cache flag, useless --max-model-len cuts), and operational breaks in LiteLLM/nginx.

Why it matters

Benchmarks sell generation speed—here that is a fraction of traffic. Teams that “buy a bigger card and hope” can pay thirty-fold latency for months without a cache-read line in the stats. For closed customer-code environments, “own stand vs subscription” must be compared on limits and audit, not list price per million tokens.

In practice

1. If you run MoE/hybrid and never see cache read — check vLLM version and explicit --enable-prefix-caching.
2. Price completed tasks and cache-read share, not only output tok/s.
3. Cross-check gateway billing with engine counters (author saw 96.9% vs 97.3%).
4. Do not shorten --max-model-len “to free VRAM” without measuring — in the write-up it only cut context.
5. Plan KV-pool ceilings and prefix eviction as active users grow.

Takeaway

For agent coding inside a perimeter, the winner is not the smartest slide-deck model but prefix cache, token observability, and honest subscription math. Rare production numbers worth reading before the next GPU purchase.