DeepSeek released DeepInfer, an open-source inference framework for its models. It supports DeepSeek-V4.1-Flash. The framework includes optimized kernels and parallelism strategies. It offers DSpark, confidence-scheduled verification, and SWA bounded replay. Asynchronous scheduling and batch-invariant inference are enabled by default. Deterministic sampling and structured outputs are also supported. It runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. A KV cache system and disaggregation are required for full deployment. Performance benchmarks are provided for prefill and decode. The code is licensed under Apache 2.0.
What it unlocks
Running DeepSeek models at scale with optimized kernels and parallelism strategies.
- 15.3k GitHub stars
- 1.5k forks
- 4 SM100 GPUs for prefill benchmarks
- 64 SM100 GPUs for decode benchmarks
What you need to act on it
- Linux OS
- GCC >= 13.0
- Python 3.12
- CUDA 13 for NVIDIA GPUs
- CANN 9.2 for Huawei Ascend NPUs
- NVIDIA SM100 or Huawei Ascend 950 hardware
Sources