DeepInfer Is Open Source: Run DeepSeek’s Inference Engine

DeepSeek just open-sourced the actual inference engine running their production API. Not a demo. Not a research prototype. DeepInfer is what moves 155 trillion tokens per day through DeepSeek’s clusters — and it is now yours under Apache 2.0. There are real constraints: it only runs DeepSeek-V4.1-Flash, requires NVIDIA Blackwell or Huawei Ascend 950 hardware, and ships without an HTTP server. Here’s what you get, how to set it up, and when you should actually use it over vLLM or SGLang.

What DeepInfer Actually Is

This is not a wrapper around vLLM. It is the kernel stack that powered DeepSeek’s API from day one. DeepSeek had previously open-sourced individual components — FlashMLA, DeepGEMM, DeepEP — but DeepInfer bundles them together with the full production logic: speculative decoding via DSpark, confidence-scheduled token verification, and TileLang-based custom GPU kernels.

The standout property is batch-invariant inference. Most inference engines produce slightly different outputs for the same input depending on the batch it lands in. DeepInfer does not. Run the same prompt at temperature zero a thousand times across a thousand different batches and you get the same output every time. For downstream systems that depend on reproducible outputs — evals, deterministic pipelines, regression testing — this matters significantly.

The companion library deepseek-recipe handles the codec layer: it converts Messages, Chat Completions, and Responses requests into the token sequences the model actually sees, using the same logic as DeepSeek’s official API. If you care about bit-for-bit parity with DeepSeek’s cloud outputs, deepseek-recipe is what gets you there.

Who Can Actually Run It

This is the detail that most coverage glosses over: DeepInfer requires NVIDIA SM100 (Blackwell) GPUs — specifically B200, GB300, or B300 cards — with CUDA 13, or Huawei Ascend 950 NPUs with CANN 9.2. H100 is not on the list. Neither is A100 or any consumer GPU.

If you are on Hopper-generation hardware or below, DeepInfer is not for you today. Use vLLM or SGLang instead — both have claimed benchmark parity with DeepInfer on some configurations anyway (SGLang posted up to 1.04x prefill throughput on GB300/B300 on October 24). DeepInfer’s advantage is in the proprietary kernel optimizations and batch invariance, not raw throughput alone.

Setup: Two Paths

Prerequisites: Linux, GCC 13 or newer, Python 3.12, and the right GPU with the right driver stack.

Build from source (NVIDIA):

git clone https://github.com/deepseek-ai/DeepInfer.git
cd DeepInfer
python3.12 -m venv .venv && source .venv/bin/activate
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

Prebuilt wheel (faster path): A community-maintained prebuilt is available on HuggingFace compiled for SM100, CUDA 13.2, Python 3.12:

huggingface-cli download inferencekey/deepinfer-sm100-cu132 \
  --local-dir /workspace/deepinfer-prebuilt
bash /workspace/deepinfer-prebuilt/install.sh /workspace/di-venv /workspace

Once installed, run offline inference. The first run auto-downloads the deepseek-ai/DeepSeek-V4.1-Flash-1017 checkpoint into your HuggingFace cache:

python examples/offline_inference.py

# Or point to a local copy to skip the download
export DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017
python examples/offline_inference.py

What It Cannot Do Yet

DeepInfer is an offline inference library. It is not a serving stack. These gaps are significant if you’re planning production deployment:

• No HTTP server. There is no built-in OpenAI-compatible endpoint. You will need to build a serving layer on top — FastAPI, a vLLM fork, or wait for the community wrappers to stabilize.
• No KV cache persistence. Each inference call starts cold. Persistent prefix caching across requests is not included.
• No load balancing. Routing, autoscaling, and multi-replica coordination are your problem.
• One model only. DeepSeek-V4.1-Flash-1017 is the only checkpoint officially supported right now.

The serving layer — KV cache management, prefill/decode disaggregation, replica balancing — is what stayed behind. DeepSeek flagged these as gaps the community will fill or they will add later.

DeepInfer vs vLLM vs SGLang: When to Use Each

The decision is straightforward once you frame it correctly:

• vLLM: Multi-model serving, OpenAI-compatible API, broad hardware support, production-ready HTTP server. The default choice for anything that isn’t specifically DeepSeek V4.1-Flash on Blackwell hardware.
• SGLang: Prefix-heavy RAG workloads, multi-turn conversations, scenarios with high prefix overlap (60%+). SGLang’s RadixAttention delivers 3-5x prefill latency improvement over vLLM in those conditions.
• DeepInfer: You have Blackwell or Ascend 950 hardware, you are specifically running DeepSeek-V4.1-Flash, and you need batch-invariant deterministic outputs or want to study production-grade MoE inference internals.

Apache 2.0 and What It Signals

The Apache 2.0 license is permissive by design: use it commercially, fork it, embed it in a product. The model weights carry their own terms, but the inference framework itself is clean. That is a deliberate signal from DeepSeek about their infrastructure philosophy.

DeepSeek’s open-source trajectory has been systematic: individual kernels first (FlashMLA, DeepGEMM, DeepEP), then the full framework (DeepInfer), with the serving layer presumably next. The pattern suggests a production-grade HTTP serving layer is coming — either from DeepSeek or from the community building on what just dropped.

If you have the hardware, the framework is on GitHub now. If you’re on H100 or earlier, bookmark it and check back when the Blackwell install base grows — or when vLLM or SGLang absorb these kernels, which both projects are already moving toward.