By ByteBot
llama.cpp 0.4.0 landed September 4 with four changes that matter if you run local inference directly — not through Ollama. The headliners: lazy tensor reading for oversized embedding tables, per-slot context limits for multi-user servers, sparse flash attention via GGML 0.23.0, and video input through the existing mtmd system. Initial support for Qwen3.8-Flash-Next (the Qwen4 architecture preview) and Nemotron-3-Puzzle-75B rounds out the release.
Lazy Tensor Reading Is Now Required for Large MoE Models
The most practically urgent addition is --lazy-mode. Models with large per-layer embedding tables — specifically Qwen3.8-Flash-Next and Gemma-4 E-series — can no longer be comfortably run without it. Qwen3.8-Flash-Next carries a 97.7 GiB unquantized n-gram embedding table. Without lazy loading, that table sits resident in RAM for the entire session. With --lazy-mode, rows are read from disk on demand via mmap instead.
Three modes are available: on reads rows from disk always (requires mmap), auto enables lazy loading only for tensors larger than 4 GiB, and off preserves the old always-resident behavior. Start with auto. It is the most conservative option that still delivers the memory benefit on Qwen3.8-Flash-Next, where the embedding table is the primary RAM bottleneck. For small models, skip it — disk read latency becomes significant relative to token generation time when the tensors are small enough to fit in RAM without issue.
# Run Qwen3.8-Flash-Next with lazy mode enabled
./llama-server --model qwen3.8-flash-next-q4_k_m.gguf --lazy-mode autoPer-Slot Context Limits Fix a Real Server Deployment Gap
The llama.cpp server divides its total context pool evenly across slots. Previously, there was no way to assign a different budget to individual slots — every slot got an equal share, whether the request needed it or not. A single long-context request could overflow or starve concurrent users.
0.4.0 adds per-slot context limits. Each slot now gets its own independent context constraint, which means multi-user API servers can cap context per slot without routing requests to separate server processes. It is a genuine improvement for production server operators running shared inference endpoints.
One caveat worth stating plainly: per-slot context limits are not a security isolation boundary. Slots share the same process and memory space. If you are running a truly multi-tenant deployment where tenants cannot trust each other, use container or process-level isolation — not slot separation.
GGML 0.23.0: Sparse Flash Attention and Apple RDMA
The backend bump from GGML 0.22.0 to 0.23.0 brings two meaningful additions for specific hardware configurations.
Sparse flash attention via ggml_flash_attn_ext_set_n_kv_max targets DeepSeek-V4, GLM-5.3-Flash, and the Qwen4exp architecture. GLM-5.3-Flash already uses hybrid attention that delivers roughly a 4.4x smaller KV cache than a dense model of comparable capability — sparse attention in GGML compounds that advantage at inference time.
The Apple RDMA transport is relevant only for multi-Mac inference clusters, but it is a significant improvement there. macOS Tahoe 26.2 shipped RDMA over Thunderbolt 5, and llama.cpp now exposes this as an RPC transport. Standard RPC throughput degrades as nodes are added to a cluster; RDMA does not. Benchmarks on an M3 Ultra cluster show standard RPC dropping from 20.4 to 15.2 tok/s as nodes scale, while RDMA holds and improves — reaching 31.9 tok/s. If you run a Mac cluster, this is worth enabling.
Video Input: Promising, But Check Your FFmpeg First
Video input arrives in 0.4.0 via the mtmd multimodal system. The implementation runs FFmpeg as a subprocess rather than bundling it, which sidesteps codec licensing complications but adds a hard dependency. If FFmpeg is not installed, video input will not work.
Technically, the mtmd_bitmap_init_lazy function accepts a <__media> marker per video and expands it into decoded frames at tokenization time. The server and CLI required minimal changes to accommodate this because frame expansion happens internally before the frames reach the rest of the pipeline — each frame is treated as an image.
Practical use cases right now are research-oriented: local video Q&A, testing temporal reasoning in vision-language models, frame-level analysis workflows. This is not a feature to drop into a production pipeline today, but the architecture is clean and it works.
New Models: Qwen3.8-Flash-Next and Nemotron-3-Puzzle-75B
Qwen3.8-Flash-Next (registered internally as the qwen4exp architecture) is a 125B MoE with 6B active parameters per token, 512 experts with 10 routed plus 1 shared, and a 51B n-gram embedding layer. The lazy mode feature above is essentially a prerequisite for running it without extraordinary hardware. Note that llama.cpp optimization for the qwen4exp architecture is still in progress — performance will improve in subsequent releases. GGUF weights are available via Unsloth on Hugging Face.
NVIDIA’s Nemotron-3-Puzzle-75B-A9B also gains native support in this release.
How to Upgrade
Check the full release notes on GitHub before upgrading. The session and state format versions have been bumped for KV-cell token tracking, which means saved states from earlier versions will not load cleanly. Rebuild from source is the recommended path:
git pull && cmake -B build && cmake --build build --config Release -j$(nproc)Server operators running behind Ollama or LM Studio will need to wait for those tools to pull in 0.4.0 before the per-slot context limits and --lazy-mode flags are accessible. Direct llama.cpp users can upgrade now. The server README covers the updated slot configuration parameters in full.
