White Circle open-sourced Halo, a distributed training framework for Hugging Face models. It adds expert, context, tensor, and expert-tensor parallelism directly to Hugging Face models. No new checkpoint format or model porting is required. It delivers 2.3–2.8x the throughput of stock TRL on the same hardware. Adding a new model family takes around 100 lines of code. It supports MoE models with expert parallelism and long sequences with context parallelism. It includes fused kernels, a bf16 optimizer, and async RL loops. LoRA fine-tuning runs 1.25x faster than full fine-tuning under expert parallelism. It works on B200/B300, H100/H200, and 24 GB consumer GPUs. Runs are configured via a single YAML file. Docker images are pre-built for each GPU architecture. The framework is designed for teams operating between single-GPU and Megatron-scale setups.
What changed
Distributed training required Megatron or manual porting with new checkpoint formats.
What it unlocks
Running distributed training on Hugging Face models without leaving the Hugging Face ecosystem.
- 2.3–2.8x throughput vs stock TRL
- ~100 lines to add a model family
- 1.25x faster LoRA vs full fine-tuning
- 9,600–9,900 tok/s/GPU on gpt-oss-20b
What you need to act on it
- B200/B300, H100/H200, or 24 GB consumer GPU
- Docker and YAML configuration
Sources