By On July 1, 2026, a developer going by JustVugg published Colibri on GitHub: a 14,700-line pure-C inference engine that runs GLM-5.2, a 744-billion-parameter Mixture-of-Experts model, on a machine with 25GB of RAM and no GPU. The project hit Hacker News with 730+ points. The top comment was blunt: “This is the hacker spirit.” Fourteen thousand GitHub stars later, the project now supports five model families including a 2.8-trillion-parameter model. It deserves a closer look.
The Trick: You Don’t Need All 744 Billion Parameters at Once
Mixture-of-Experts models are large on paper but sparse in practice. GLM-5.2 has 744B total parameters, but only about 40–55 billion activate for any given token. The rest sit idle. Colibri exploits exactly this sparsity.
The engine keeps the dense backbone — roughly 9.9 GB at int4 quantization — permanently in RAM. The remaining ~370 GB of expert weights live on your NVMe drive, divided into 21,504 expert blocks. A per-layer LRU cache loads experts from disk only when the router activates them. Over time, a learning cache auto-pins frequently used experts in memory, reducing disk reads as the session warms up. The result: you need 25GB of RAM and 370GB of NVMe storage, not 744GB of RAM.
Under the hood, Colibri also applies MLA (Multi-head Latent Attention) with a 57× compressed KV cache — 576 floats per token instead of 32,768 — and uses native multi-token prediction speculative decoding to generate 2.2–2.8 draft tokens per forward pass at int8. These are not trivial implementation details; the author studied the GLM-5.2 architecture closely and implemented it correctly.
The Honest Numbers
Do not mistake this for a cloud API replacement. Performance varies sharply by hardware:
- Minimum config (25GB RAM, consumer NVMe): 0.05–0.1 tokens/second. A 100-token response takes up to 33 minutes.
- MacBook M4 Pro (48GB RAM): 0.30 tok/s
- MacBook M5 Max (128GB RAM): 2.06 tok/s
- 6× RTX 5090 workstation (251GB RAM): 5.8–6.8 tok/s
The bottleneck is NVMe random-read throughput, not sequential speed. Swapping a 1.51 GB/s drive for an 8.81 GB/s drive delivered only 2.9× the speedup — not the theoretical 5.8×. The cache hit rate matters more than raw bandwidth. More RAM translates directly to fewer disk reads, which is why the M5 Max at 128GB runs roughly 20× faster than a minimum-spec box at 25GB. A complete performance breakdown is in the repository’s benchmarks documentation.
What Is GLM-5.2, Exactly?
GLM-5.2 is not a toy model. Developed by THUDM at Tsinghua University, it was trained on 28.5 trillion tokens and supports a one-million-token context window. On several benchmarks it outperforms GPT-5.5. The MoE design uses 256 experts with 8+1 active per token, and the first three transformer layers run dense before expert routing begins. This is a frontier-grade model that, until Colibri, required a cluster to run. For a technical breakdown of the MoE architecture, MindStudio has a solid deep-dive.
Models Supported as of August 2026
Colibri has expanded well beyond its initial GLM-5.2 demo. Supported model families now include:
- GLM-5.2 — 744B total, ~40–55B active per token
- Inkling — 975B total, 41B active
- Kimi K3 — 2.8 trillion parameters, 104B active
- DeepSeek V4 Flash — 284B total, 13B active
- OLMoE — 7B total, 1B active (fastest option)
Colibri is not a generic GGUF loader. Each model family requires its own architecture-specific engine — a real limitation, but also a sign the author is building correct implementations rather than approximations. Compare this to Needle2’s approach, which went the opposite direction: a 14MB model that runs on a Raspberry Pi.
Who Should Actually Use This
The use case is narrow but real. Colibri makes sense for:
- Privacy-sensitive workloads — medical records, proprietary source code, financial data that cannot leave the machine
- Air-gapped environments — defense, regulated industries, disconnected research systems
- Overnight batch jobs — if response latency does not matter and you want zero API cost
- Researchers and tinkerers — who want to interact with frontier-scale model weights directly, without API rate limits
It does not make sense for interactive chat, latency-sensitive APIs, or any workflow where waiting 33 minutes for a response is not acceptable. For those, the cloud is the right answer. Colibri is not competing with vLLM or Ollama on their terms; it is opening a lane those tools do not occupy. For more on the local AI spectrum, see our coverage of Meta Muse Glimmer, which targets a very different use case.
Getting Started
Requirements: gcc with OpenMP, AVX2 support, 16GB+ RAM (25GB+ recommended), and ~370GB of free NVMe storage for GLM-5.2. No Python. No CUDA. No pip install. The repository is at github.com/JustVugg/colibri with quickstart documentation and community-submitted benchmark results for dozens of hardware configurations.
What Colibri Actually Signals
Colibri proves that disk-streaming MoE inference is not theoretical — it is working code that ships today. The architectural pattern (sparse activation + NVMe streaming + learning cache) will only get faster as PCIe 5.0 and 6.0 NVMe become standard. A token rate of 0.1 tok/s in 2026 is not impressive. But the same architecture running on hardware two generations from now will be routine. Colibri is the proof of concept the field needed to know this direction is worth building.
