By ByteBot
If you’ve ever priced out running LLM inference on AMD hardware or a Google TPU and then looked at what it would actually take to port your stack there, you know the answer is usually “not worth it.” Every serious inference setup today is wired to NVIDIA CUDA — which means switching chips means rewriting code. ZML, a Paris startup, shipped LLMD on July 8 — a free, open-source inference server that replaces that rewrite with a Docker tag swap.
LLMD runs LLaMA, Gemma, Qwen, and Mistral on NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI, and Apple Metal — from the same image, with the same OpenAI-compatible API. Your client code does not change. Only the image tag does.
One Command, Five Chip Families
Here is what the portability looks like in practice:
# NVIDIA CUDA
docker run -p 8000:8000 --gpus=all -e HF_TOKEN \
-it zmlai/llmd --model=hf://Qwen/Qwen3-8B
# AMD ROCm -- change the tag, keep everything else
docker run -p 8000:8000 --device=/dev/kfd --device=/dev/dri -e HF_TOKEN \
-it zmlai/llmd:rocm --model=hf://Qwen/Qwen3-8B
# Google TPU
docker run --net=host --privileged -e HF_TOKEN \
-it zmlai/llmd:tpu --model=hf://Qwen/Qwen3-8BThe API endpoint, the model path, the client library — none of that changes. If you are using the OpenAI Python SDK pointed at http://localhost:8000, it works identically across all three. That is the point.
Why Hardware Lock-In Is the Real Problem
The reason this matters is that AI inference hardware is fragmenting fast. AWS Trainium 3 is real competition on cost — but it requires Amazon’s Neuron SDK, which means your stack only runs on AWS. Google’s TPU Ironwood requires JAX. AMD’s MI300X offers excellent VRAM capacity for large models at a fraction of H100 pricing, but ROCm support in most inference frameworks is second-class at best.
The result: teams that could save 40–60% on inference costs by moving to alternative hardware cannot, because the switching cost is too high. You would be rewriting your serving layer, revalidating your model behavior, and retraining your ops team. ZML’s bet is that a chip-agnostic inference server breaks that lock-in at the infrastructure layer — before any of that becomes your problem.
What’s Under the Hood
LLMD is not a Python wrapper over CUDA kernels. ZML built their stack in Zig, using MLIR and Bazel, compiling model computation graphs into native binaries for each target platform via OpenXLA. This is a fundamentally different architecture than vLLM or TGI, which are Python-first and NVIDIA-first by design.
The feature set is solid for an inference server: continuous batching, paged attention, tensor-parallel sharding, tool calling, and Prometheus metrics. The container clocks in at 2.4 GB — compact enough for fast autoscaling. The API is OpenAI-compatible, so dropping LLMD behind an existing codebase requires no client-side changes.
The Caveats You Need to Know
ZML is calling this a technical preview, and they mean it. LLMD is single-GPU only right now — no multi-GPU sharding. The maximum batch size is 16. There are no published throughput benchmarks against vLLM (which hits 793 tokens per second in Red Hat’s testing). The supported model list is limited to LLaMA, Gemma, Qwen, and Mistral.
If you need production inference today, you are still on vLLM for NVIDIA or Ollama for local dev. LLMD does not replace either yet. What it does is give platform engineers a real option to test cross-chip portability without building the abstraction layer themselves.
ZML raised $20 million in July 2026 — from 20VC, Kima Ventures, LocalGlobe, and Puzzle Ventures — which suggests they have runway to get LLMD to production-grade. The roadmap presumably includes multi-GPU support, broader model coverage, and performance parity with vLLM on NVIDIA.
Should You Try It?
If you are a platform engineer managing LLM inference costs, yes — spin it up today in a test environment on AMD or TPU hardware and see what the portability story looks like in practice. The 2.4 GB image pulls fast. The OpenAI API compatibility means your existing test suite runs against it without changes. The problem LLMD is solving is real enough that you want to understand the tool before you need it.
If you are running production inference, stay on vLLM for now. But watch ZML closely. The cross-chip inference server gap is real, it is getting more expensive to ignore, and nobody else has shipped a credible answer to it yet. The repo is public — worth a star and a bookmark.
