Prime Intellect rebuilt how its training stack copies model weights to inference servers. The new path uses NIXL, a direct memory access library from Nvidia's Dynamo team, plus its own ModelExpress. Copying a 1.6 TB GLM-5.2 policy now takes under four seconds instead of 86 seconds. The system traces how vLLM lays out weights at runtime, so no mapping is written by hand. Transfer of one group of weights overlaps compression of the previous group. Most of the remaining delay is vLLM waiting for all workers to agree to pause. Prime Intellect changed that agreement to happen every wave rather than every 32. It is working with the vLLM team to check whether this hurts serving throughput. The older method required a fixed set of processes, which blocked fault tolerance and auto-scaling. Auto-scaling on the new foundation is not yet built.
What changed
Weight sync from trainer to inference took 60-90 seconds using NCCL broadcasts.
What it unlocks
Syncing a trillion-parameter policy to inference workers in under four seconds, without a fixed set of processes.
- 86.1s to 3.9s per weight sync
- 1.6 TB policy transferred
- 45 GB/s of 50 GB/s NIC peak
What you need to act on it
- prime-rl and vLLM
- RDMA-capable GPU cluster with per-GPU 400Gb/s NICs
- per-wave pause consensus in vLLM
Sources