Researchers proposed disaggregated quantization to specialize formats for prefill and decode. Removing activation quantization on decode improved accuracy on decode-heavy tasks. Training separate compute-native prefill weights accelerated prompt processing. It matched or exceeded weight-only inference accuracy at 2-3-bit decode. An NVFP4 prefiller improved 1-bit accuracy by 32.5 points on MMLU-Pro. It also improved 1-bit accuracy by 35.3 points on MMMU-Pro. Offloaded disaggregated prefill streamed weights from SSD. This delivered a 1.78x time-to-first-token speedup at 8K prompt length. The method was validated on models up to 2.8T parameters.
What changed
LLM quantization used the same precision and format for both prefill and decode phases.
What it unlocks
Using different quantization formats for prefill and decode phases to improve accuracy and speed without increasing inference cost.
- 32.5 points 1-bit accuracy gain on MMLU-Pro
- 35.3 points 1-bit accuracy gain on MMMU-Pro
- 1.78x time-to-first-token speedup at 8K prompt length
- 2-3-bit decode weights
What you need to act on it
- Qwen 3 or Gemma 3 model
- vLLM or llama.cpp serving environment
- SSD storage for offloaded disaggregated prefill
Sources