By ByteBot
llama.cpp v0.6.0 shipped October 5 with four changes that actually matter: speculative decoding is 3.4x faster on Apple Silicon, a new /v1/systemone endpoint enables local decision models for agent routing, the C batch API got deprecated, and a 320B hybrid text-vision model now runs natively. Here’s what changed and what you need to do.
Speculative Decoding on Apple Silicon — Fixed
If you’ve been running a draft model on Metal and noticed the performance was worse than plain decoding, that wasn’t your imagination. The old Metal backend handled speculative batches with mat-vec kernels, which scale linearly with every additional draft token row. More tokens in the speculation window meant more work per decode step, making the approach net-negative on Apple Silicon.
v0.6.0 fixes this with new few-row MMA (Matrix Multiply Accumulate) kernels that use the GPU’s actual matrix multiply hardware. They kick in at a threshold — as low as 2 rows for most quantization formats — and deliver what the GPU was designed to do.
The numbers are hard to argue with. On an M3 Ultra, speculative decoding went from 32.1 tok/s to 110 tok/s — a 3.4x increase from a software update. A subsequent commit extended the same kernels to all 26 weight formats on Metal, including BF16 and MXFP4, pushing mat-mul up to 4.4x faster in the best case. The cutoff is MTLGPUFamilyApple7+, meaning M2 and newer benefit. For more details on the benchmark methodology, see the full breakdown at HuggingNews.
This matters beyond raw throughput. Draft-model workflows — where a small model proposes tokens that a larger model verifies — are now the right default on Apple Silicon instead of the wrong one. MoE architectures like Qwen4Exp and Gemma4 also benefit via new Metal flash attention kernels and MADVISE-based prefetching for large embedding tables.
Decision Models: Local Agent Routing Without Generation
v0.6.0 adds a /v1/systemone endpoint implementing the SystemOne decision model API. A decision model is not a text generator — it answers typed questions about a given text in a single forward pass and returns a probability distribution over a fixed list of options. No tokens generated, no hallucination surface.
The use case is agent infrastructure: routing incoming requests, selecting which tool to call, validating whether input matches a schema, moderating content. Anything where you need a reliable classification answer and don’t want to burn a full generation call to get it.
Five open GGUFs are supported, all published by ggml-org on Hugging Face. Julia-1 weighs in at 144M parameters under Apache 2.0 — small enough to run alongside a 70B inference server without meaningful overhead. Kev-4B and Lev occupy the middle ground. OpenJev at 27B adds vision support under CC BY-NC 4.0. See the full list of supported decision model GGUFs for download links and quantization options.
Starting a decision model server takes one command:
llama serve -hf ggml-org/Kev-4B-GGUFThen POST your routing request:
curl -s http://localhost:8080/v1/systemone -H "Content-Type: application/json" -d '{
"state": "User: My payment failed and I want a refund.",
"question": "Which team handles this?",
"options": ["billing", "support", "engineering", "sales"]
}'The response is a probability distribution over those four options — no generated text, just scores. OpenAI launched a Decisions API earlier this year for the same general problem. This is the local, open-source equivalent that runs on your hardware.
The llama_batch API Is Deprecated — Update Before v0.7.0
If you have code that calls llama_batch_get_one(), llama_batch_init(), or llama_batch_free() directly, those functions are deprecated as of v0.6.0. They still compile and run — with deprecation warnings — but the project plans to remove them in v0.7.0.
The replacement is llama_batch_ext with llama_process(). The change was necessary to support mixed token and embedding inputs in a single batch, which MTP draft models and deepstack architectures require. All official examples, llama-server, and the speculative decoding tools were already migrated in this release. For wrapper libraries, the migration is mechanical — the deprecation warnings point to the correct replacements.
If you’re using llama-cpp-python or another higher-level binding, check whether your version has been updated. Most maintained wrappers track upstream closely. If yours hasn’t released a v0.6.0-compatible build yet, pin to the last 0.5.x release until it does.
Also in This Release
GLM-5.3-Flash, a 320B hybrid text-and-vision model (18B active via MoE), now runs natively in llama.cpp — the first 320B model in the ecosystem. Multi-sequence runs require --kv-unified. GGUF quantizations are available from Unsloth and AtomicChat on Hugging Face if you have the hardware.
The built-in web UI at localhost:8080 was overhauled to add a Hugging Face Hub browser. You can search for and download GGUF models directly from the UI without leaving the server page. Downloads land in the standard Hugging Face cache directory, shared with Python tooling, so the same model won’t be pulled twice across applications.
Who Should Upgrade
Upgrade now if you run speculative decoding on Apple Silicon — the performance regression is fixed and the speedup is real. If you’re building agentic workflows, the decision model API is worth testing; Julia-1 at 144M is nearly free to run. If you maintain any code calling the old batch API directly, start the migration: the deprecation window closes at v0.7.0.
For everyone else, this is a standard upgrade. The full v0.6.0 release notes on GitHub cover every commit. No regressions have been reported in the week since release.
