vLLM 0.31: Fix Breaking Changes Before You Upgrade

vLLM 0.31.0 landed October 5 with 717 commits from 307 contributors — the largest release the project has shipped by commit count. Two things demand your attention: a set of breaking changes that will silently wreck production deployments, and a new preload daemon that eliminates the cold-start restart penalty you’ve been working around for years.

Breaking Changes: Check These Before You Upgrade

Three changes in 0.31 break existing configurations without loud errors. Audit before running pip install --upgrade vllm.

tokenizer_mode=“slow” Is Gone

The tokenizer_mode=“slow” flag has been removed entirely. It behaved identically to “hf” since Hugging Face Transformers v5 — the team finally made it official. Remove it from configs and startup scripts. Use “hf” explicitly if needed, or omit the flag entirely.

Per-Request Multimodal Kwargs Are Now Gated

This one bites multimodal deployments. Per-request mm_processor_kwargs and media_io_kwargs now return an error by default — these kwargs can affect inference behavior in ways that aren’t safe to expose to arbitrary clients without operator intent.

Two fixes: if clients legitimately need per-request multimodal settings, add --trust-request-mm-kwargs to your server startup command. Otherwise, move those settings server-side into --mm-processor-kwargs. Server-level configuration and offline LLM() calls are unchanged.

fp8 Quantization Shorthand

quantization=“fp8” is now treated as shorthand for fp8_per_tensor. Minor, but worth testing quantized model startup after upgrading if you’re using explicit fp8 strings in config.

Fast Restart: The Feature That Changes Production Operations

vLLM cold starts on large models cost 30 to 90 seconds of downtime per restart. On Kubernetes, that means maintaining minimum replica counts and careful HPA tuning just to avoid restart-induced unavailability. The new preload daemon attacks this directly.

vllm preload launches a per-GPU daemon process that keeps post-quantized, TP-sharded weights resident in GPU memory. When an engine restarts, instead of reloading from disk and re-running quantization, it connects to the daemon via Unix domain socket and maps the weights through CUDA IPC handles — zero-copy by default. Restart time drops to seconds.

# Run once — keep this process alive
vllm preload --model meta-llama/Llama-3.2-1B-Instruct --tensor-parallel-size 4

# Start your engine with weight cache enabled
vllm serve meta-llama/Llama-3.2-1B-Instruct   --tensor-parallel-size 4   --load-format ipc_cache

In the default zero_copy mode, the daemon must stay alive for the engine’s lifetime — one copy of weights per GPU, no extra memory footprint. If the daemon is unavailable or weights don’t match the engine config, vLLM falls back to disk load automatically. A copy mode also exists, where the engine clones weights into its own GPU memory and releases the daemon’s cache, trading memory efficiency for independence.

0.31 extends preload with data parallelism support, MTP draft model support, a /health endpoint, and a readiness wait. Also experimental: vllm snapshot create/restore using CRIU to checkpoint a fully initialized TP1 engine and restore without reinitialization — the direction toward sub-second restarts.

Speculative Decoding Reaches Model Runner V2

Model Runner V2 is the new default execution path in vLLM. Before 0.31, speculative decoding required the legacy execution path, which locked out MRV2 users. That gap is closed.

0.31 adds draft-model speculative decoding to Model Runner V2, along with the new LiLiCorr drafter. LiLiCorr reranks global candidate token lattices using target model embeddings, full-vocabulary log probabilities, and draft hidden states. In benchmarks, it holds the highest throughput in 63 of 72 measured configurations against competing parallel-draft methods. DSpark adaptive verification also ships for Gemma4 deployments.

DeepSeek-V4 Fill-in-the-Middle Support

If you’re self-hosting DeepSeek-V4 for coding workflows, 0.31 adds native FIM support through the standard /completions endpoint. Pass a suffix alongside your prompt and vLLM renders the request into DeepSeek’s FIM token format before inference — no client-side workarounds or custom tokenizer logic required. Full release notes on GitHub.

Upgrade Checklist

• Search configs and startup scripts for tokenizer_mode=“slow” — remove it
• Multimodal deployments: add --trust-request-mm-kwargs or move kwargs server-side
• Test quantized model startup if using quantization=“fp8” explicitly
• Kubernetes operators: set up the preload daemon before the next model deploy