Edge0 shipped a 145-billion-parameter open-source model on November 12 that runs on a single NVIDIA GPU — no server rack required. Edge0-145B uses a sparse mixture-of-experts architecture that activates only around 10 billion parameters per inference step, combined with SSD-streamed expert weights and a predictive router. The result is 29 GB peak memory, 80+ tokens per second, and coding benchmark scores that beat both Qwen3.8-Flash-Next and DeepSeek-V4-Flash-0731 locally.
Not Compression — Architecture
The common assumption with on-device AI is that you are paying a quality tax for the privilege of local inference. Edge0-145B pushes back on that. A dense 145B model requires upward of 400 GB of memory — the kind of setup you find in a data center, not under your desk. Edge0 design gets the memory footprint down to 29 GB without gutting model capacity.
How? Sparse mixture-of-experts. The model has 512 expert networks across 48 layers, but only 10 experts activate per token at inference time. Each group of 64 experts also shares two large weight matrices, with each expert holding only its unique parameters. Total parameters: 145 billion. Active parameters per step: about 10 billion. The rest stay on disk.
The Three Pillars That Make It Fast
Sparsity alone does not get you to 80 tokens per second if you are constantly stalling on disk reads. Edge0 inference engine — open-sourced at github.com/Edge0-AI/edge0 under Apache 2.0 — adds three mechanisms to close that gap.
SSD expert streaming. Expert weights are stored as int4 tensors on SSD and memory-mapped on demand using the OS page cache with an LRU eviction policy. Active experts load into a staged fixed-slot double buffer — only what is needed, when it is needed.
Prerouter. This is the clever part. A lightweight head on each layer predicts the next layer expert routing one token ahead. The prediction drives the prefetch, so SSD reads overlap with the compute for the current layer. When execution reaches the next layer, the experts are already in memory. The staging cost disappears into the pipeline. The technique comes from Edge0 research paper, The Other Half of the Memory Wall, on serving large MoEs from SSD with trained routing prediction.
Recover-LoRA. The base model is frozen at 4-bit integer precision. LoRA adapters, distilled from a full-precision teacher, run as a parallel branch during inference to recover quality lost to quantization. The result of training natively at 4 bits is a model that holds up better than a post-hoc quantized equivalent.
Benchmark Numbers
Edge0 reports these on vendor hardware — independent reproduction matters and has not happened at scale yet. That said, the numbers are specific enough to be meaningful:
| Model | DeepSWE 1.1 | Terminal-Bench 2.1 | Peak Memory |
|---|---|---|---|
| Edge0-145B | 64.6 | 86.52 | 29 GB |
| Qwen3.8-Flash-Next | 58.7 | 83.3 | — |
| DeepSeek-V4-Flash-0731 | 54.4 | 82.7 | ~87 GB |
On an RTX 5090 D with 24 GB VRAM, Edge0-145B hits 121.2 tokens per second agent decode and 3,554.9 tokens per second on 16K prefill (Q3_K format). On a compatible high-memory laptop it delivers 80+ tokens per second generation. The comparison to DeepSeek-V4-Flash is notable: similar benchmark territory, 3x the memory footprint on DeepSeek side.
What You Need to Run It
macOS users should temper expectations for the 145B: it currently requires Linux or Windows with an NVIDIA GPU and CUDA 13. The smaller Edge0-35B runs on Apple Silicon. For the flagship model, you need at minimum a 24 GB VRAM GPU. Two formats are on HuggingFace:
- Edge0-145B-q3k — 40.81 GiB, Q3_K quantization
- Edge0-145B-NVFP4 — 47.79 GiB, NVIDIA FP4 format
The fastest path to running it:
# Via llama.cpp (easiest cross-platform option)
llama serve -hf Edge0/Edge0-145B-q3k
# Or via native edge0-infer
edge0-infer serve --model /path/to/Edge0-145B-q3k
edge0-infer chat --model /path/to/Edge0-145B-q3kThe server exposes OpenAI- and Anthropic-compatible APIs, so existing tooling works without modification.
Why This Release Lands Differently
Local LLM deployment has had a soft ceiling. Qwen3.8-27B at 27 billion parameters has been the practical high-water mark for most consumer hardware setups. The jump to 145B is not incremental — it is the scale where reasoning and coding capability start behaving differently. Edge0-145B DeepSWE score of 64.6 versus the Qwen3.8-27B neighborhood reflects that: at 145B sparse, you get materially better performance on engineering tasks.
The timing matters too. Privacy regulation, cloud cost pressure, and the growth of agentic workloads that make hundreds of LLM calls per session are all pushing developers toward local inference. Edge0-145B, fully open-sourced under Apache 2.0 by founder Samuel Zeng, makes a credible case that the tradeoff between quality and local deployment is narrowing faster than most expected.
The model weights, inference engine, and architecture paper are all public. The official launch post covers the full technical depth, and the prerouter paper goes deep on the SSD streaming mechanism. Try it, benchmark it independently, and see whether the numbers hold.
