vLLM introduced a push-based NIXL connector for disaggregated inference. The prefill instance now writes KV directly into decode memory over RDMA. This removes the router relay and decode-side setup from the critical path. Push mode delivered up to 3x better TTFT and 30% better TPOT in benchmarks. The feature was built on AWS Trainium and contributed to vLLM as accelerator-agnostic. It overlaps producer and consumer work to reduce latency. A background thread handles transfer preparation on the prefill side. The decode instance registers target blocks up front. Push mode selects the decode instance earlier than pull mode. Users can choose between push and pull connectors based on workload needs.
What changed
The default NIXL connector used pull-based KV transfer, where the decode instance pulled KV blocks from the prefill instance after prefill completion.
What it unlocks
Starting decode token generation sooner by removing post-prefill coordination steps from the critical path.
- 3x better TTFT
- 30% better TPOT
- 1P1D deployment on Amazon EC2 P5en
- Qwen3-32B model
What you need to act on it
- disaggregated prefill/decode inference in vLLM
- NVIDIA Inference Xfer Library (NIXL) connector
- RDMA-capable interconnects like AWS EFA
- vLLM configured with NixlPushConnector
Sources