SGLang, Qwen and NVIDIA implemented NVFP4 KV cache for long-context inference. NVFP4 packs four-bit values with two-level scaling to cut KV storage. The format uses 56% of the space of FP8 for the same tokens. Decode throughput rose 26–30% at matched concurrency. It rose 37–78% when more requests fit in memory. Accuracy drops were under 1.6 percentage points on tested benchmarks. Prefill performance changed little. Agentic workloads benefit from retaining longer histories on GPU. The work targets agent sessions with many conversation rounds. The implementation covers initial prefill, chunked prefill and decode paths.
What changed
KV cache used FP8 or BF16 formats with higher memory footprint.
What it unlocks
Running more concurrent long-context or agentic inference jobs on the same GPU memory.
- 56% of FP8 storage for NVFP4 KV
- 26–30% decode throughput gain at matched concurrency
- 37–78% decode throughput gain at higher concurrency
- 0.08–1.60 percentage point accuracy drop vs FP8
What you need to act on it
- NVIDIA Blackwell GPUs
- SGLang runtime
Sources