Contents
In brief
Colibri is an experimental C inference engine that runs GLM-5.2, a 744-billion-parameter MoE model, on a consumer machine with 25GB of RAM and no GPU. That headline needs an important qualification: the author reports just 0.05–0.1 tokens per second on the development system.
The project is compelling because it shifts most routed expert weights to NVMe storage while keeping the always-used portion of the model in memory. It is an architecture lesson, not a recipe for a responsive local chat service.
What happened
According to the project write-up, Colibri keeps the dense layers, shared experts, and embeddings resident in roughly 9.9GB of RAM. Its 21,504 routed experts, around 370GB in int4 form, live on an NVMe drive and are loaded when the model router selects them.
That is possible because an MoE model does not use every parameter for every token. The article estimates that GLM-5.2 activates about 40B parameters per token. Colibri turns that sparsity into a storage hierarchy: memory holds the stable working set, while the drive supplies selected experts.
The engine also includes compressed persistent conversation state, expert caching, asynchronous prefetching, and multi-token prediction. Those features try to reduce disk waits, but do not remove the fundamental cost of fetching model data from storage.
Why it matters
Colibri makes a useful distinction between total model size and simultaneously required model state. A model with hundreds of billions of parameters may be technically runnable on modest RAM when its architecture is sparse and its access pattern can be managed.
For engineers, this is a concrete example of working around a memory limit with routing, quantization, and caching. For product planning, it is equally useful as a reminder that local execution and interactive performance are separate requirements.
| Dimension | Author's reported setup | Consequence |
|---|---|---|
| Resident memory | about 9.9GB | feasible on a 25GB machine |
| Model storage | about 370GB on NVMe | fast local storage is essential |
| Generation speed | 0.05–0.1 tokens/s | unsuitable for normal chat UX |
| Implementation | small, dependency-free C engine | approachable for systems study |
In practice
The best fit is work where a result can arrive later: offline summarization, batch document analysis, local evaluation, or privacy-sensitive processing that should not leave a controlled machine. It is not a replacement for an API serving many concurrent users.
If you want to test this type of system, treat storage performance as a first-class dependency:
- Reserve several hundred gigabytes on a fast NVMe drive; network-mounted storage undermines the design.
- Benchmark random reads and memory use with your own prompts instead of relying on a published token rate.
- Put long-running jobs behind a queue and describe the expected completion time honestly.
- Validate model provenance, licensing, and downloaded weight integrity before handling sensitive input.
The same techniques can matter in faster runtimes too. A cache that learns which experts are hot and a prefetcher that anticipates routing can reduce data movement wherever an MoE model is deployed.
Takeaway
Colibri does not make a 744B LLM convenient on every laptop. It demonstrates that a sparse model can be split across memory tiers and run locally when slow responses are an acceptable trade-off.
That makes it a valuable open implementation to study: one focused on expert routing, KV-cache compression, quantization, and I/O rather than an oversized framework. For a production decision, start with the latency budget—not the parameter count.