Contents
In brief
Once an open LLM becomes part of a production system, the question is no longer simply which model ranks highest. Teams need reproducible inference settings, a valid output contract, clear visibility into each model call, and hardware-aware model choices.
An Habr article based on production experience with llama.cpp, Gemma, and Qwen distils eight lessons. Its central argument is simple: infrastructure should enforce the properties that a business process cannot leave to a probabilistic model.
What happened
The authors run open models for ticket classification, document processing, and tool-using conversational systems. Their initial experiments started with Ollama, but production requirements eventually led them to llama.cpp, where inference settings and GPU memory use can be controlled more directly.
That control has a cost: the team must maintain its own configuration. In return, it can pin versions, tune tensor placement for a given GPU, and repeat a known configuration rather than relying on hidden defaults. The exact speed-up will vary by model and hardware, so the useful lesson is to benchmark the actual deployment.
They also moved away from looking for one universal model. In their setup, Gemma is used to compose natural, client-facing text, while Qwen handles stepwise reasoning, classification, and tool calls. A pipeline of specialised stages produced more reliable outcomes than making a single model do every kind of work.
Why it matters
Benchmark leaderboards answer only part of a production question. A model must also work in the target language, fit into available VRAM at a useful quantisation level, stay within latency limits, and behave well on the company’s own examples.
Prompt portability is another easy-to-miss risk. Models use different chat templates and role conventions. Passing a template designed for another model often produces no error message; quality simply declines. A model change should therefore be treated as a configuration and evaluation change, not as a drop-in replacement.
Agent frameworks have a similar trade-off. They can accelerate exploration of evolving, branching workflows, but can hide retries and model calls in a stable process. When every call matters, explicit orchestration code may be easier to observe, test, and operate.
In practice
- Start with the task. Define language, quality, latency, context, and memory constraints before choosing a model.
- Split model responsibilities. Evaluate extraction, classification, reasoning, and user-facing writing as separate stages.
- Version inference configuration. Keep model weights, quantisation, chat template, temperature, and memory allocation under source control.
- Enforce structured output. llama.cpp can constrain generation with GBNF grammars or JSON Schema instead of relying on a “return JSON” instruction.
- Validate semantics too. A grammar guarantees shape, not that the values are accurate or that a response was not cut off by a token limit.
- Tune temperature per stage. Deterministic extraction and decision steps need different settings from customer-facing prose.
- Use frameworks selectively. Retain them where complex state and branching are changing quickly; prefer transparent code for fixed pipelines.
- Measure every change. A new model, prompt, or template is a new system configuration that needs regression tests.
Takeaway
Open models can offer data control, predictable high-volume economics, and independence from an external API. They do not remove the engineering work around serving, prompts, response formats, and observability.
For a short prototype, the most convenient runner is often the right choice. For a system that drives a business process, production readiness comes from making the important behaviour explicit, measured, and enforced outside the model.