DeepInfer: High-performance inference framework for DeepSeek models

You can now deploy DeepSeek models for inference using an open-source framework that offers optimizations for performance and scale.

DeepSeek AI has open-sourced DeepInfer, a high-performance inference framework designed for their DeepSeek models. This framework is intended for large-scale production services and offers optimized kernels like FlashMLA, DeepGEMM, and TileKernels. It supports features such as attention DP/balanced CP, MoE EP + EPLB parallelism, DSpark verification, and batch-invariant inference. DeepInfer is compatible with NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, and it is built on technologies including vLLM, SGLang, and TensorRT-LLM. The release includes offline inference capabilities with support for models like DeepSeek-V4.1-Flash-1017, with configuration options for performance tuning. Building the framework requires a Linux environment with GCC, Python 3.12, and either CUDA 13 for NVIDIA or CANN 9.2 for Ascend. The project aims to provide the core features necessary for serving large models at scale.

The open-sourcing of DeepInfer signals a growing trend where leading AI labs are releasing their internal inference frameworks to foster wider adoption and community development, moving beyond just model releases. This move by DeepSeek AI, leveraging components from established frameworks like vLLM, suggests a competitive landscape where performance and efficiency in model deployment are becoming key differentiators, directly impacting the cost and feasibility of running advanced AI models.

This release offers developers and organizations a high-performance solution that has already been proven in production environments, potentially reducing the barrier to entry for deploying large-scale AI services. By providing specialized kernels and advanced parallelism techniques, DeepInfer aims to unlock significant performance gains, enabling more efficient utilization of hardware resources and lowering operational expenses for inference.