By ByteBot
llama.cpp 0.7.0 landed on October 15 with three changes that matter for anyone running MoE models locally: a GPU expert cache that pushes decode throughput up to 2.2x, a new NVIDIA-derived speculative decoding technique, and a default server port change from 8080 to 9931 that will silently break your setup if you don’t act. The MoE cache alone makes this a mandatory upgrade for anyone running DeepSeek, Qwen3, or Meta Muse models that overflow their VRAM.
The MoE Cache: What Changed and Why It’s Fast
Mixture-of-Experts models route each token through only a small subset of their total parameters. That’s why a 30B-active-parameter model like Qwen3-30B-A3B can run at reasonable speed despite having 110B total weights — most of those weights aren’t touched per inference step. The problem arrives when the model exceeds your VRAM: llama.cpp offloads the unused experts to CPU RAM, and every token requires a fresh host-to-GPU transfer for each expert it needs. At scale, that back-and-forth kills throughput.
The new --moe-cache-mib flag fixes this with an LRU cache. Hot experts — the ones your model hits repeatedly — stay resident in GPU memory. Only cache misses trigger uploads. For a 30B MoE model on an RTX 5090 paired with --cpu-moe, this pushed decode from 30.8 to 67.8 tokens per second — a 2.2x jump. On a free Tesla T4 (16 GB), the same Qwen3-30B-A3B went from 22.6 to 43.2 t/s. Even an RTX 3060 running Gemma-4-26B-Q4 gained 29% with a 7,200 MiB cache.
# Allocate 8 GiB of GPU memory for the expert cache
llama-server -m qwen3-30b-a3b-q4_k_m.gguf --moe-cache-mib 8192Set --moe-cache-mib as high as your available VRAM allows after the model’s non-expert layers are loaded. Multi-GPU setups are supported — the cache spans all devices. The community benchmarks from DEV.to show consistent gains across a range of hardware, including a near-doubling on a free Colab T4.
Breaking: The Default Server Port Is Now 9931
If you run llama-server behind a reverse proxy or have any application hardcoded to port 8080, this upgrade will break your setup without warning. The new default port is 9931 — leetspeak for “GGML”, which is as good a reason as any. Bare-metal deployments with nginx, Caddy, or Traefik proxying to localhost:8080 are directly affected. Docker users whose container-to-host port mapping was already explicit are not.
# Keep the old port behavior
llama-server -m model.gguf --port 8080Or update your reverse proxy upstream to port 9931. Either way, verify before you upgrade in any environment where downtime matters.
LiliCorr Speculative Decoding
llama.cpp 0.7.0 also adds LiliCorr, a speculative decoding improvement from NVIDIA’s Nemotron research team. The core problem with DFlash — llama.cpp’s diffusion-style drafter — is that it generates draft tokens independently per position. Tokens look plausible in isolation but often form incoherent sequences, dragging down acceptance rates.
LiliCorr addresses this by running the entire candidate lattice through a lightweight two-layer Transformer in one forward pass. Self-attention spans all candidates across all positions, scoring joint coherence rather than per-position likelihood. The NVIDIA paper reports 7–19% higher acceptance length with only about 3% additional latency per block. In llama.cpp 0.7.0, it works best with GLM5-Next via the MTP speculative decoding path.
Model Manager and New Models
The web UI ships a long-overdue model manager that auto-discovers GGUF files in your configured directory, loads models on demand when first requested, and uses LRU eviction to keep up to four models resident simultaneously. Switching between models without restarting the server is now practical.
New model support includes K2 Horizon (dense and MoVA variants), Prism Bonsai 2 27B, EmbeddingGemma-2 (text, vision, and audio), LiquidAI d1/d1-omni decision models, pplx-decider, MiniCPM-V 4.7, and Nemotron Super 3.5 VL. ggml is updated to v0.27.0.
Upgrade Now
If you’re running any MoE model that partially overflows your VRAM — and most popular open models in 2026 are MoE — the speedup is real and takes one flag to unlock. Check the full release notes for the complete changelog before upgrading in production.
# macOS via Homebrew
brew upgrade llama.cpp
# From source with CUDA
git pull
cmake -B build -DGGML_CUDA=on
cmake --build build --config Release
