By ByteBot
The vLLM team published benchmarks on August 23 showing speculative decoding achieves up to 2.87x throughput on AMD Instinct MI300X and MI355X GPUs — numbers that directly challenge the assumption that AMD is second-tier for LLM inference. Five distinct drafting methods were tested across Gemma 4, Qwen3, and Kimi-K2.5 model families, proving the technique works on AMD hardware and doesn’t require Nvidia CUDA. For developers already running self-hosted LLMs on AMD, this is free performance. For teams choosing inference infrastructure, it changes the economics calculation.
Draft Fast, Verify Faster
Speculative decoding exploits a fundamental inefficiency in standard LLM inference: GPUs are memory-bandwidth-bound. The chip’s compute units sit largely idle, waiting on HBM reads between each token generation step. Standard autoregressive inference generates one token per forward pass. Speculative decoding replaces that with a draft-and-verify loop: a small, fast draft model proposes K candidate tokens; the large target model verifies all K in a single parallel forward pass. Tokens that match are accepted; the first mismatch triggers a reset. This fills the GPU’s idle compute with verification work that costs almost nothing when proposals are accurate.
In practice, enabling it through vLLM requires three extra lines of configuration. According to the official vLLM AMD speculative decoding blog post, AMD MI300X’s 192GB HBM3 at 5.3 TB/s memory bandwidth is well-suited for this technique — more bandwidth means faster draft model execution and higher acceptance throughput.
from vllm import LLM, SamplingParams
llm = LLM(
model="google/gemma-4-26b",
speculative_config={
"method": "dflash",
"num_speculative_tokens": 5,
},
tensor_parallel_size=1, # MI300X's 192GB fits 26B without sharding
)
sampling_params = SamplingParams(temperature=0.0, max_tokens=512)
outputs = llm.generate(["Explain speculative decoding in one sentence:"], sampling_params)
Five Methods, One Default
The vLLM benchmarks tested five distinct drafting approaches, and they aren’t interchangeable. EAGLE-3 is the safe default: a lightweight autoregressive draft head that plugs into the target model, reuses internal features across multiple transformer layers, and has been merged into vLLM, SGLang, and TensorRT-LLM main branches as of early 2026. Start here unless your model supports something better. For full technical detail on EAGLE-3 speculative decoding on AMD Instinct GPUs, the vLLM team published a dedicated deep-dive in July.
Native MTP (Multi-Token Prediction) is better when available. Models with built-in MTP — Qwen3.5 and several Gemma 4 variants — include drafting capability natively, eliminating the overhead of loading a separate draft model. On Qwen3.5-122B-A10B, native MTP achieved 2.20x throughput. DFlash and its sibling DSpark achieved the highest absolute numbers — DFlash generates draft tokens in parallel rather than sequentially, reaching 2.87x on Gemma-4-26B. However, DFlash has stricter model compatibility requirements. Use EAGLE-3 as your baseline; switch to native MTP or DFlash if your model supports it and you need the extra headroom.
Related: Gimlet Labs $300M: The Multi-Silicon Inference Bet on Post-GPU AI
AMD Is Viable Now — With Caveats
The honest picture: AMD MI355X achieves 90-95% of H100 throughput for standard LLM inference at batch sizes 64-128. MI300X cloud rental pricing runs $1.50-2.50/hr versus H100 SXM5 at $2.39/hr, translating to roughly 44% lower cost-per-token at equivalent throughput. According to Spheron’s ROCm vs CUDA 2026 benchmark analysis, AMD’s memory bandwidth advantage — 5.3 TB/s versus H100’s 3.35 TB/s — makes it especially competitive for memory-bound large-model inference.
However, AMD is not viable if you need TensorRT-LLM or FlashAttention 3. Neither has a ROCm equivalent as of September 2026. TensorRT-LLM can triple throughput over baseline vLLM on Nvidia — meaning an H100 with TRT-LLM can outperform an MI300X with speculative decoding on optimized high-throughput serving. The decision is now a software question, not a hardware question: if your stack is vLLM or SGLang with open-source models, AMD is cost-competitive. If you’re invested in TRT-LLM, stay on Nvidia.
The Catch: Concurrency Kills the Gains
The 2.87x headline is for batch size 1. At batch size 16-64, gains drop to 1.2-1.5x. At batch size 128+, speculative decoding provides minimal benefit — the GPU is no longer memory-bandwidth-bound when handling many concurrent requests, removing the idle-compute opportunity the technique exploits. As the BentoML LLM inference handbook notes, speculative decoding excels for interactive and low-concurrency workloads: internal tools, per-user sessions, agent pipelines processing one thread at a time. Don’t promise your team 3x production throughput if you’re running a public API at scale.
Key Takeaways
- Speculative decoding now works on AMD MI300X/MI355X via vLLM with up to 2.87x throughput at batch size 1 — the technique is no longer CUDA-exclusive
- EAGLE-3 is the safe default method; use native MTP if your model supports it (Qwen3.5+); use DFlash for maximum throughput when model compatibility allows
- AMD is cost-competitive for vLLM/SGLang deployments with open-source models, but TensorRT-LLM and FlashAttention 3 still have no ROCm equivalent — this remains a real limitation
- Gains diminish sharply at high concurrency; speculative decoding is most valuable for interactive and low-to-medium concurrency workloads, not high-traffic production APIs
- MI300X’s 192GB HBM3 eliminates tensor parallelism for models up to ~70B, reducing inter-GPU latency — a separate advantage that compounds the speculative decoding benefit
For developers already on AMD hardware, enabling speculative decoding through vLLM’s speculative_config is the highest-ROI configuration change available today. For teams evaluating GPU providers, the AMD cost story is now credible — provided your stack doesn’t depend on TensorRT-LLM.
