Researchers proposed four techniques to bound memory peaks in long-context MoE training. PipelinedLLEP caps tokens per dispatch chunk. Ring-DTP circulates activations around a ring for vocabulary projection. Selective checkpoint offload keeps one tensor per boundary in CPU memory. OffloadStreamAdamW pipelines CPU Adam updates. All four preserve exact loss and gradients. Tests showed up to 59.3% dispatch peak reduction without throughput loss. Vocabulary projection peak fell by 86.6%. Offloaded optimizer steps ran 2.05× faster. Composed, they enabled 1M context length training. Reach improved 8–32× over FSDP2 baselines. Throughput rose up to 10.4×.
What changed
Long-context MoE training failed when any single memory peak exceeded device capacity.
What it unlocks
Training MoE models at 1M context length without memory failures.
- 59.3% MoE dispatch peak reduction
- 86.6% vocabulary projection peak reduction
- 2.05× faster offloaded optimizer step
- 1M context length achieved
What you need to act on it
- MoE model between 120B and 667B parameters
- GPU cluster with fixed working set at launch
Sources