By ByteBot
Ollama 0.40 has been quietly landing on Macs for two weeks, and most developers haven’t noticed the most important part: MLX is now the default inference backend on Apple Silicon, no configuration required. If you have 32GB of unified memory and pull a supported model, you’re already running on the faster path. Decode throughput roughly doubled on the same hardware. This is worth an ollama pull.
Why the Speed Doubled
The previous Ollama backend used llama.cpp with Apple’s Metal API. Metal works, but it treats Apple Silicon like any discrete GPU: tensors live in system RAM, get copied to GPU memory, processed, then copied back. That copy overhead compounds on every token generation step.
MLX eliminates the copies. Apple Silicon’s unified memory gives the CPU and GPU access to the same physical pool. Model weights, the KV cache, and activations all coexist without transfer overhead. Token generation is memory-bandwidth bound, so removing that bottleneck directly translates to faster output — and that’s exactly what the numbers show.
The Benchmarks
On an M5 Max running Qwen3.5-35B-A3B with NVFP4 quantization, Ollama’s official numbers are:
- Prefill (prompt processing): 1,154 → 1,810 tokens/sec (+57%)
- Decode (token generation): 58 → 112 tokens/sec (+93%)
- Int4 quantization workloads: +131%
On M4 Max hardware with smaller models, the pattern holds. Qwen2.5 7B goes from 28 to 54 tokens/sec. Llama 3.3 70B moves from 12 to 23 tokens/sec. Across the board, you’re looking at approximately 2x decode throughput on the same chip.
One nuance worth understanding: M5 Neural Accelerators deliver 3–4x faster prefill compared to M4 (compute-bound), but only 1.2x faster decode (memory-bandwidth bound). Don’t let M5 marketing numbers obscure what’s actually happening on your hardware.
Three Things to Do Right Now
The upgrade is straightforward, but there’s one gotcha: existing models don’t automatically switch to MLX format. You need to re-pull them.
# Step 1: Update Ollama
brew upgrade ollama
# Step 2: Re-pull models to get MLX-format weights
ollama pull qwen2.5:7b
ollama pull gemma4:9b
# Step 3: Verify MLX is active
tail -f ~/.ollama/logs/server.logOllama 0.40.2 handles auto-upgrades for already-downloaded models where possible, but pulling fresh is the clean path. No API changes are needed — same endpoints, same clients, same commands you were already using.
What Runs on MLX (and What Doesn’t)
The 0.40 release added MLX support for the Qwen3 family (3.5, 3.6, 3.8), Gemma4, and the new decision models. Most Llama 3.x variants are in testing. If you pull a model whose architecture isn’t MLX-compatible yet, Ollama falls back to the Metal path silently — nothing breaks, you just don’t get the speed gain.
Macs with 8GB or 16GB of unified memory also fall back to Metal automatically. There’s no hard cutoff enforced in Ollama’s code, but 32GB is the practical floor for 35B-class models. A 32GB M4 Pro will fit Qwen3.5-35B but tightly (~20GB model + ~12GB cache). For long-context sessions, 48GB is the comfortable target.
Decision Models: The Feature Developers Are Overlooking
Buried in the 0.40 release notes is a genuinely new capability: decision models via the /v1/systemone endpoint. These are non-autoregressive — instead of generating a token stream, they return probabilities or scores in a single forward pass. Nimble (9B) averaged 91ms per decision on an M5 Max running locally. That’s faster than a generative call and requires no output parsing.
ollama pull nimble
# POST to http://localhost:11434/v1/systemoneSupported question types are choice (probability per option), noul (true/false probability), and score. The practical use cases are clear: ticket triage, agent routing, content classification, and anywhere you’re currently paying for a generative call just to get a category label. Ollama’s decision models post covers the full API spec.
Apple Silicon Is a First-Class Inference Platform Now
The bigger picture here isn’t the 2x number. It’s that Apple’s MLX framework has become the default path for local LLM inference on Mac, not a workaround. When Ollama — the tool that drives most developer local AI workflows — ships MLX as the out-of-the-box default, that’s a signal. The “Metal-compatible GPU” framing is dead. Developers with 32GB+ Macs now have legitimate cloud-competitive inference at zero marginal cost per token.
If you’ve been waiting to see whether Apple Silicon was serious about AI, Ollama’s backend choice is your answer. Check the Ollama releases page for the full 0.40.x changelog.
