AWS introduced a push-based KV transfer mode for vLLM. It removes the serialization of the pull model by overlapping producer and consumer work. It moves transfer setup off the decode instance to a background thread. The change reduces time-to-first-token and time-per-output-token. Benchmarks on P5en instances with EFA showed up to 3x better TTFT. TPOT improved by up to 30 percent under high load. The feature was built on Trainium and contributed to vLLM as accelerator-agnostic. It works with GPUs and Trainium chips. The push connector coordinates decode, prefill, and a background transfer thread. Registration happens on the decode side before prefill completes. The prefill side issues the transfer once both registration and KV are ready. A background thread handles NIXL WRITE to keep the main loops free. The design allows different block layouts on each side.
What changed
vLLM used pull-based NIXL connector for KV transfer in disaggregated inference.
What it unlocks
Starting decode sooner by moving KV transfer setup off the critical path.
- up to 3x better TTFT
- up to 30% better TPOT
- 1,744 ms TTFT at 8 QPS and 16k input
- 540 ms TTFT at 8 QPS and 16k input
What you need to act on it
- disaggregated prefill/decode inference with vLLM
- NVIDIA Inference Xfer Library (NIXL) connector
- AWS EFA or supported RDMA interconnect
- Amazon EC2 P5en or Trainium instances
Sources