DeepSeek has released DeepInfer under the Apache 2.0 license, describing it as the high-performance inference framework that powers its own large-scale production services. The public release covers offline inference for DeepSeek-V4.1-Flash, with the 1017 checkpoint as the default, and it runs on two hardware targets: NVIDIA SM100-class Blackwell GPUs and Huawei Ascend 950 NPUs. The timing was deliberate. The repository landed on the first day of PyTorch Conference, and it passed fifteen thousand GitHub stars within roughly a day.
The feature list reads like a map of where serving efficiency for large mixture-of-experts models actually comes from. It includes optimized kernels plus model and sampler runtimes; attention data parallelism with balanced context parallelism; expert parallelism for the mixture-of-experts layers with expert-parallel load balancing; a speculative path called DSpark with confidence-scheduled verification; bounded replay for sliding-window attention; an asynchronous, zero-overhead batch scheduler; batch-invariant inference and deterministic sampling; and structured outputs. The published benchmarks were measured with 4 SM100 GPUs for prefill and 64 for decode, each inside one scale-up domain. DeepSeek credits vLLM as the original base, with ideas drawn from SGLang and TensorRT-LLM.
What is missing matters as much as what shipped. DeepSeek says a complete service also needs a KV cache system, disaggregation of prefill and decode (most of the engine functionality for it is already present), and load balancing across data-parallel replicas, and none of those are in this release. Latent Space called these the crown jewels: a compact KV cache with lifetimes beyond eight hours, which underpins prompt caching of up to 72 hours, compared with the 12 hours OpenAI just announced. Latent Space also headlined production figures of about 155 trillion tokens per day per cluster and roughly 22 million dollars of daily revenue at around 80 percent margin. Interconnects cited outside estimates as high as 90 percent gross margin. Treat both as estimates rather than audited numbers.
The strategic read is straightforward. Leading labs are widely understood to run far more efficient inference internally than open tools like vLLM or SGLang provide. As far as Interconnects could tell, this is the first time a top lab has published such a stack. Inference providers serving DeepSeek weights can now close much of that gap, which should push third-party prices for DeepSeek models down, or margins up. First-class Ascend support also fits DeepSeek's reported co-design work with Huawei. The release comes the same week as a Department of Homeland Security order barring DeepSeek from federal civilian procurement, and a Chinese Commerce Ministry proposal requiring pre-approval before frontier models are shared abroad.
- Latent Space framed it as the open-source answer to closed inference decacorns and stressed the withheld KV cache, disaggregation and load balancing.
- Interconnects called it one of the biggest open-source AI artifacts in months and set it against new US and Chinese controls on open models.
- The Information emphasized that the code exposes how DeepSeek keeps compute costs low at scale.
- Hacker News discussion centered on the repo itself; it passed 15,000 stars within about a day.