DeepInfer: DeepSeek Open Sources Its Production Inference Engine Without the Serving Layer

By Indie Kings | October 22, 2026

Updated October 22, 2026: DeepSeek has published DeepInfer, the inference framework it says powers its own large-scale production services, as an Apache-2.0 repository on GitHub. The public code landed on 20 October 2026, supports exactly one model, and ships without three of the components a complete serving stack needs.

Image: GitHub repository card for deepseek-ai/DeepInfer, DeepSeek's open-sourced inference framework.

What DeepInfer is

The one-line description is careful: DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services, and what has been open-sourced is the offline inference code with the core features needed for serving at scale. Not the whole serving platform. The README lists what a complete inference service needs and then states which of those the release does not include.

The repository is young in a way that is worth noting before anything else. It was created on 16 October 2026, and the initial commit is dated 20 October 2026, attributed to Jiashi Li. By the time of writing it has 19,653 stars, 2,085 forks and 192 open issues, which is a rate of attention that has almost nothing to do with how much code there is. There are four commits total, from three contributors: beginlner with two, wx-csy with one and hkeee21 with one. The most recent, on 21 October from Shaoyuan CHEN, bumps the DeepEP submodule to a specific revision and drops an SSH to HTTPS rewrite in the submodule handling.

That gap between stars and commits is the first thing to understand about this release. It is not a mature codebase opened late. It is a working production component published early, and the community response has been to watch it rather than to change it.

The three pieces that were left out

The README has a short section titled "Required for a complete inference service (not included in this release)". It lists three things, and each one is a load-bearing part of a real deployment.

Component	Status in this release	What it does in a full service
KV cache system	Not included	Stores attention keys and values across requests so long conversations and shared prefixes do not recompute
VE, prefill and decode disaggregation	Most required engine functionality already implemented, per the README	Splits the compute-heavy prefill stage from the latency-sensitive decode stage onto different pools
Load balancing across data-parallel replicas, prefill and decode	Not included	Decides which replica receives which request when several copies of the model are running

The phrasing on the middle row is the interesting one. Disaggregation is not simply absent; the README says most of the required engine functionality is already implemented, which implies the engine side is there and the orchestration around it is what a user would have to supply. The other two are stated flatly as not included.

Anyone who has run vLLM or SGLang in production will recognise what this means in practice. The missing KV cache layer is the difference between a system that handles one conversation at a time well and one that serves many concurrent users without recomputing the same prefix. The missing load balancer is the difference between a single replica and a fleet. DeepInfer gives you the engine and a worked example of the model, and leaves the multi-tenant infrastructure to you.

What is in the box

What is included is a fairly specific set of optimisations, and the README exposes most of them as switches rather than burying them.

Feature	How it is controlled	Notes from the README
DSpark with confidence-scheduled verification	`DEEPINFER_ENABLE_MTP_VARLEN=1`, with `DEEPINFER_USE_NEXTN_DECODE` for maximum draft length	Draft length defaults to 5; draft tokens are verified according to confidence estimates
Decoder SWA bounded replay	`DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact` and `DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140`	Replays only a bounded suffix during prefill; the retained suffix must cover the sliding-window attention window plus maximum backtracking length
Asynchronous scheduling	Enabled by default	Overlaps CPU batch scheduling with device execution
Batch-invariant inference	`DEEPINFER_BATCH_INVARIANT=1`	NVIDIA only; keeps results identical across different batch compositions
Deterministic sampling	Batch-invariant plus `DEEPINFER_ENABLE_MTP_VARLEN=0` and a fixed `seq_seed`	NVIDIA only; identical outputs for identical input tokens and sampling parameters
Structured outputs	`so_args` passed to `SamplingParams`	Regex, JSON schema or EBNF grammar constraints

The two determinism features deserve a note because they are the ones that separate a demo from something a paying customer can be served on. Batch-invariant inference means the result of a request does not depend on what else happened to be in the batch with it, which is a real problem for mixture-of-experts models because routing decisions vary with batch composition. DeepSeek restricts both features to NVIDIA hardware, so on the Ascend path you do not get them.

The kernel stack underneath is the part DeepSeek has been publishing piece by piece over the past two years, and DeepInfer pulls it together as submodules rather than vendoring copies: FlashMLA, DeepGEMM, DeepEP, DeepSelect, plus Ascend variants of DeepEP and DeepGEMM, and in-tree TileLang kernels under the model executor's layer directory. Parallelism support is listed as attention data parallelism with balanced context parallelism, and mixture-of-experts expert parallelism with EPLB.

The acknowledgement section is unusually honest for a project this size. DeepInfer was, per its own README, originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. That is worth holding next to the feature list: several of the properties above exist in those projects too, in different form, and what DeepSeek has added is the layer specific to how its own model wants to be served.

Two accelerators, one model

The support matrix is the shortest part of the README and the most revealing.

Supported models: DeepSeek-V4.1-Flash. That is the only entry. Supported hardware: NVIDIA SM100 GPUs, and Huawei Ascend 950 NPUs. Prerequisites on the NVIDIA side are Linux, GCC 13.0 or newer, Python 3.12 and CUDA 13; on the Ascend side the same Linux, GCC and Python requirements with CANN 9.2 instead of CUDA. The build is selected by exporting DEEPINFER_ACCELERATOR as either nvidia or ascend before running the build script.

Publishing first-class Ascend support alongside NVIDIA is not a routine choice for a Western-facing release, and it is consistent with the way DeepSeek has handled its previous kernel releases, which shipped Ascend variants as separate repositories. The practical consequence for a reader is that the same engine targets two very different programming models, and the NVIDIA-only tag on batch invariance and deterministic sampling is the clearest sign that the two paths are not at feature parity yet.

The default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017 on Hugging Face, downloaded into the local cache on first run. Setting DEEPINFER_HF_MODEL_DIR points it at a local copy instead, which matters for anyone running without outbound access. The examples directory documents four launch shapes: 8-way expert parallelism, 2-way expert with 8-way data parallelism, 2-way expert with 8-way context parallelism, and the defaults.

How it is measured

This is where DeepInfer is more forthcoming than most inference frameworks. The repository ships an offline batch evaluation harness, and its README specifies the benchmark setup precisely enough to argue with.

Benchmark	Tasks	Samples per task	Reported metric
AIME 2026	30	32	Pass@1, mean accuracy over 32 samples
HLE	2,500 (2,158 text, 342 image)	1	Pass@1, single-sample accuracy
GPQA Diamond	198	8	Pass@1, mean accuracy over 8 samples

Generation and scoring are deliberately separate stages. The generation stage submits every request in a single batch and writes raw outputs to disk; the scoring stage reads them and produces metrics, so scoring can be repeated with a different judge without regenerating anything. The harness does not download datasets, and for HLE the upstream dataset is gated, so the reader has to accept terms on Hugging Face before the pipeline will run.

Performance is published as two Pareto charts rather than a table: prefill throughput against latency for non-cached input, and decode output throughput against per-user speed. The README states the measurements were taken on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, and points at a sweep script for reproduction. There are no absolute tokens-per-second numbers in the README text, so anyone quoting a figure for this release is reading it off a chart.

What the repository actually contains

The tree is smaller than the feature list suggests, which is consistent with a release that leans on submodules. It holds 398 entries, of which 270 are Python files, 18 are C++ headers, 15 are C++ sources and 10 are shell scripts. By top-level directory the split is 223 files under the main Python package, 33 under csrc, 20 tests, 18 examples, 17 evaluation files and 8 under a cluster topology package.

The Python package is organised the way a serving engine usually is: distributed, engine, entrypoints, hugging face integration, model executor with an EPLB subdirectory, NPU ops, structured output, a vision encoder processor and a worker. The native side is split into an engine with input metadata, a graph timer and utilities, which is the shape you would expect from a project that started on vLLM and kept its C++ extension boundary.

The evaluation directories are named after the benchmarks, and the tests carry platform markers: ci for GPU core tests, ci_npu for NPU-only tests, and both applied to tests that must run on either. The contribution guide asks for Ruff on Python and a clang-format script on C++ and CUDA, and keeps core tests deliberately lightweight.

FAQ

Is DeepInfer a replacement for vLLM?
Not from what it publishes. The README says it was originally built primarily on vLLM, and the feature set is tuned to one model family. It is the engine DeepSeek runs for DeepSeek-V4.1-Flash, not a general-purpose server for arbitrary checkpoints.

Which models does it support?
One: DeepSeek-V4.1-Flash. The default checkpoint is DeepSeek-V4.1-Flash-1017, downloaded from Hugging Face on first run unless DEEPINFER_HF_MODEL_DIR points at a local copy.

What hardware do I need?
NVIDIA SM100 GPUs, or Huawei Ascend 950 NPUs. The NVIDIA path needs CUDA 13 and the Ascend path needs CANN 9.2, and both need Linux, GCC 13 or newer and Python 3.12.

Can I run it as a complete serving stack?
No. The README states that a KV cache system, load balancing across data-parallel replicas and full prefill/decode disaggregation are not included, though it says most of the required engine functionality for disaggregation is already implemented.

Is it licensed for commercial use?
It is Apache-2.0, which is a permissive licence, so the licence itself is not the barrier. The barrier is what is missing: you would be building the serving layer around the engine.

Why does batch invariance only work on NVIDIA?
The README lists both batch-invariant inference and deterministic sampling as NVIDIA only without explaining the reason. Neither is available on the Ascend path as published.

Bottom Line

DeepInfer is the most useful thing DeepSeek has open-sourced since its kernels, and the useful part is not the model support or the benchmark harness. It is that a team building a serving layer can now read the decode-side tricks a frontier lab actually ships: confidence-scheduled draft verification, bounded replay for sliding-window attention, an asynchronous scheduler on by default, and a documented attempt at making batch composition stop changing results. What it does not ship is just as informative. No KV cache, no data-parallel load balancer and no complete disaggregation means this is an engine release and not a platform release, and anyone who expects to point it at traffic on day one will be writing the missing three pieces themselves. The Ascend 950 path, published alongside NVIDIA support while batch invariance and deterministic sampling stay restricted to NVIDIA, is the clearest signal of where the two builds stand relative to each other. Read it for the engine, and treat the 19,653 stars as curiosity about DeepSeek rather than an endorsement of the code, which is four commits old.

Sources

DeepInfer repository README, .gitmodules, CONTRIBUTING.md, eval/README.md and examples/README.md, deepseek-ai/DeepInfer, accessed 22 October 2026
GitHub repository metadata, commit history, contributors and recursive tree for deepseek-ai/DeepInfer, accessed 22 October 2026
DeepInfer citation block, DeepSeek-AI, 2026