By ByteBot
A solo developer just ran a 48 GB language model on a 32 GB laptop and pulled 37 tokens per second out of it. For context, llama.cpp gets 2.8 tok/s on the same hardware with the same model. That is a 13x gap, and it comes down to one architectural decision: stop pretending your SSD is not fast enough to be an inference medium.
Janas, written in C by Maurizio Cammalleri, hit the Hacker News front page this week. It runs Mixture-of-Experts (MoE) models larger than your available RAM by streaming expert weights directly from NVMe with O_DIRECT reads, bypassing the OS page cache entirely. The result: the definition of “too big to run locally” is no longer determined by your RAM. It is determined by your SSD bandwidth.
Why MoE Models Are Different
If you have wondered why a 48 GB model does not require 48 GB of active compute, this is the answer. MoE architecture divides a model into dozens or hundreds of specialized sub-networks called experts. For each token generated, a small gating network selects only two to four of those experts to activate. The rest sit idle.
A 48 GB model with 10 percent active weight per token only needs roughly 5 GB in memory at any given moment. The trick is that you do not know in advance which experts will be called. Janas solves this by treating experts like OS memory pages: frequently-called ones stay cached in RAM, quiet ones get evicted and re-streamed from disk on demand. In practice, Janas reports an 86.9 percent expert cache hit rate running a 142 GB model on a 62.5 GB machine. The hot experts stay warm. The rest cost a disk read — and on a modern NVMe drive at 5–7 GB/s, that read is fast enough to keep pace with token generation.
What Makes It Faster Than mmap
llama.cpp already supports disk-backed models via mmap. The difference is control. With mmap, the OS decides what to cache. With Janas, the engine makes that decision with expert-level awareness — it knows the difference between attention weights (always needed) and MoE expert blocks (needed sometimes), and it caches accordingly.
O_DIRECT reads bypass the page cache completely, preventing memory thrashing when the model substantially exceeds RAM. The engine fills available memory with its expert cache and reads the rest off disk at full NVMe speed, controlled entirely in application space. Janas also supports speculative drafting: a small draft block predicts the next several tokens, and the full model validates them in one pass. On Qwen3-Next-80B, that lifts write speed from 24 tok/s to 37 tok/s.
The Benchmarks
All numbers below are author-reported, measured on an Intel Core Ultra 9 185H with 32 GB RAM, Q4_K_M quantization, over 9–11 October 2026. Independent verification has not been published yet.
- Qwen3-Next-80B (48.4 GB): Janas 37 tok/s vs llama.cpp 2.8 tok/s write; 62 vs 16 tok/s read
- Qwen3-30B-A3B (18.6 GB): Janas 32 tok/s vs llama.cpp 15 tok/s
- Gemma-4-26B-A4B-it (18.3 GB): Janas 18 tok/s vs llama.cpp 12 tok/s
The claim is a clean sweep across all 13 models tested. That said, the Hacker News thread notes that llama.cpp’s mmap was not optimally configured for the comparison, and Q4_K_M is a lossy quantization. Take the absolute numbers as directional, not definitive.
How to Try It
Janas requires either a pre-converted .jns model file from the author’s Hugging Face account (Qwen3 and Gemma4 variants are available) or a GGUF you convert yourself:
bin/x86_64-linux/gguf2jns model-Q4_K_M.gguf model-flat.jns
bin/x86_64-linux/jns_planes model-flat.jns model.jnsThe Hugging Face call for testers has the quickest start path. Be aware: the CLI and API change frequently. Only the .jns file format is stable.
The Honest Picture
Janas is an early experiment by one developer. The benchmarks are self-reported. The code changes several times per week. There is also a pending llama.cpp PR implementing the same technique natively — if it merges, llama.cpp gains expert-aware SSD streaming without the format lock-in and project instability that come with Janas today.
None of that makes the concept wrong. The architectural case for expert streaming is solid, the cache hit rates suggest it works in practice, and the benchmark gap is too large to dismiss as noise. Whether Janas itself sticks around or gets absorbed into llama.cpp, the constraint for local MoE inference has shifted: RAM is no longer the ceiling. Your NVMe is.
