DeepInfer: The Inference Framework Behind DeepSeek's Production Serving

The Engine That Already Runs DeepSeek in Production

DeepInfer is the inference framework that powers DeepSeek's large-scale production services, and this open-source release is the offline-inference half of it: the core features needed for serving at scale, without the surrounding cluster machinery. It serves DeepSeek-V4.1-Flash (default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017) on two very different accelerators, NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, from one codebase whose backend is selected at build time by DEEPINFER_ACCELERATOR. The license is Apache-2.0, and the acknowledgments place it honestly in the ecosystem: originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM.

What makes the release interesting is not the feature list but the fact that it is the production engine itself, stripped down to what can run offline. The README is explicit about the boundary: a KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas are required for a complete inference service but are not included, with the note that most of the disaggregation engine functionality is already implemented. Everything between the tokenizer and the sampler is here.

A C++ Scheduler Under a Python Engine

The split is visible in the tree. deepinfer/ is Python: llm_engine.py, the model executor with its input-metadata builders (yoco.py, dflash.py), the worker cache engine, the vision-encoder path under ve/, and structure_output/ with its compiler, mask and matcher. csrc/ is C++ and CUDA: scheduler.hpp defines SchedulerOutputs (scheduled sequences, prompt-run flag, batched-token count, per-sequence seqlens_q, clone and zero state lists) and an EngineType enum with NORMAL, PROMPT and COMPLETION modes that mirrors deepinfer/engine/engine_type.py, the three engine types a prefill/decode-disaggregated deployment asks for.

Scheduling is asynchronous by default: CPU-side batch construction overlaps the previous step's device execution, which the README calls a zero-overhead batch scheduler. The benchmark harness in examples/ shows how seriously the steady state is protected. Prefill and decode are profiled in separate sections of one run; the decode profiler attaches only after the measurement window, because profiler exit costs a residual 7-11% slowdown at batch 1; and intervals that collide with a MoE load-balancing stats flush are dropped from the averages rather than smoothed over.

Parallelism Shaped Like the Model: DP, CP, EP and EPLB

Attention is parallelized with data parallelism and balanced context parallelism, while the mixture-of-experts backbone runs expert parallelism with EPLB, DeepSeek's expert load balancer. Redundant experts are configured through DEEPINFER_OVERRIDE_NUM_EXTRA_EXPERTS (default 32), and the engine treats that value as a lower bound: routed plus redundant experts are rounded up until divisible by the expert-parallel size, with the backbone and the DFlash draft head rounded separately. The deep_cluster/ package carries the hardware and topology descriptions that placement decisions need, and tests/distributed/test_expert_shuffle.py pins the shuffling behavior.

The offline entry point exposes the same axes: examples/offline_inference.py accepts --ep 8, --ep 2 --dp 8 or --ep 2 --cp 8, and the benchmark derives the expert-parallel size from WORLD_SIZE and the visible device list, requiring --cp to divide it.

DSpark: Speculative Decoding Scheduled by Confidence

Decoding is accelerated by DSpark, DeepSeek's draft-token scheme, plus confidence-scheduled verification: confidence estimates decide when a draft sequence gets verified, instead of verifying on a fixed cadence. Two environment variables gate the path. DEEPINFER_ENABLE_MTP_VARLEN=1 turns on the variable-length multi-token-prediction path, and DEEPINFER_USE_NEXTN_DECODE caps the draft length (default 5). The scheduling logic lives in csrc/engine/mtp_varlen.hpp with per-platform TileLang implementations in schedule_mtp_varlen.py, and the YoCo equivalence harness can compare confidence scores bitwise on a DFlash+YoCo checkpoint, which is as direct a probe of the draft cache as the repository offers.

The benchmark accounting follows the same idea: decode throughput is reported as average steps per second times batch size times (1 + nextn) divided by the context-parallel degree, under full-acceptance accounting, so the headline number states its assumption instead of hiding it.

SWA Bounded Replay: Prefill That Reads Only a Suffix

V4.1's decoder uses a sliding-window attention pattern, and DeepInfer exploits it at prefill with bounded replay: instead of recomputing the whole prompt, the decoder replays only a bounded suffix. DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact with DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 are the default V4.1 settings, and the constraint is stated plainly: the retained suffix must cover the sliding-window window plus the maximum backtracking length. The prefill Pareto curve in the repository is measured with exactly this configuration, at 131,072 tokens of context with decoder bounded replay at 140 tokens.

Because truncation changes what the attention mask sees, the repository ships a two-level equivalence check in examples/yoco/. L1 compares SWA pivots, visible keys, store slots and compressed key ranges on a single device in seconds, without loading weights. L2 runs teacher-forced logprobs on eight devices and compares them element by element, and with a DFlash+YoCo checkpoint additionally compares the MTP confidence bitwise.

Reproducibility as a Feature: Invariance, Determinism, PPL Baselines

Three modes turn the engine from fast into testable. Batch-invariant inference (DEEPINFER_BATCH_INVARIANT=1, NVIDIA only) keeps model results identical across different batch compositions. Deterministic sampling adds DEEPINFER_ENABLE_MTP_VARLEN=0 and a fixed seq_seed in SamplingParams so identical inputs and sampling parameters produce identical outputs. Structured outputs constrain generation to a regex, a JSON schema or an EBNF grammar through so_args, compiled and enforced by deepinfer/structure_output/.

The regression suite treats numerics the same way. Offline runs compare decode perplexity sample by sample against tests/models/offline_decode_ppl.json, keyed by platform and workload rather than by model directory, and multi-node baseline refreshes write under a cross-node file lock so two nodes never overwrite each other. The long-context case (16,389 input tokens, 4,096 output tokens, nextn=5) is part of the full pytest run.

The Numbers, and What They Cost

The published Pareto curves are measured within a single scale-up domain: prefill on 4 SM100 GPUs, decode on 64. Prefill (EP4DP4, 384+32 experts, 131k context with bounded replay) moves from about 60,000 non-cached tokens per second per GPU at a 4,096-token chunk and roughly 70 milliseconds, to about 89,500 at a 65,536-token chunk and roughly 730 milliseconds.

Decode (EP64DP64, 384+64 experts, nextn=3 fixed, per-token acceptance 75%) trades per-user speed against aggregate throughput, from about 325 tokens per second per user at batch 8 to about 26,000 tokens per second per GPU at batch 256, where per-user speed is still around 103 tokens per second.

On the accuracy side, eval/ runs offline batch evaluation for AIME 2026 (30 tasks, 32 samples each), HLE (2,500 tasks, 2,158 text and 342 image) and GPQA Diamond (198 tasks, 8 samples each), reporting Pass@1 with generation and scoring as separate stages so a different judge can rescore without regenerating. Datasets are not downloaded by the scripts; each must be obtained upstream and converted to JSONL first.

Prerequisites are current-generation: Linux, GCC 13 or newer, Python 3.12, CUDA 13 on NVIDIA and CANN 9.2 on Ascend, with build.sh vendoring the operator stack (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and the in-tree TileLang kernels). For anyone serving DeepSeek-class mixture-of-experts models at scale, DeepInfer is the rare case of a production inference engine published together with its benchmark methodology, its equivalence tests and its numeric baselines in the same repository.