Ant Group and the vLLM community open-sourced Humming. It is a kernel library for quantized inference. It supports NVIDIA GPUs from Turing onward and T-Head Zhenwu 810E. It handles many weight and activation quantization formats for dense and MoE models. It uses a unified representation to separate quantization from execution. It compiles kernels on the fly for each workload. It shares CUDA components across formats and architectures. It tunes execution based on device parameters. It outperforms vLLM Marlin, FlashInfer and DeepGEMM on many workloads. It cuts kernel compile time from 23.35 seconds to under 2 seconds on H200. It supports hidden Blackwell Tensor Core formats not exposed in PTX. It is available at github.com/vllm-project/humming.
What changed
Quantized inference relied on separate, dedicated kernels for each format and architecture combination.
What it unlocks
Running quantized models with a single, unified kernel library across many GPU types and quantization schemes.
- 23.35s to 1.82s kernel compile time on H200
- 2–4s to compile all supported kernel sizes
- 3–4× speedup at small batch sizes for W4A16
- 671B total parameters in DeepSeek-V3
What you need to act on it
- CUDA-compatible GPU from Turing onward or T-Head Zhenwu 810E
- quantized model in a supported export format
Sources