White Circle open-sourced Halo, a distributed training framework for Hugging Face models. It adds expert, context, tensor and expert-tensor parallelism directly to Hugging Face models. No porting or new checkpoint formats are required. It delivers 2.3–2.8x the throughput of stock TRL on the same hardware. Adding a new model family takes around 100 lines of code. It supports agentic and async RL, on-policy distillation and embeddings learning. It uses fused kernels, a bf16 optimizer and packing to speed up training. It works with PyTorch-native parallelism like FSDP2 and DTensor. It excludes pipeline parallelism due to GPU idle time. It integrates with SGLang for rollout generation in multi-turn RL. Docker images and YAML configs define and launch runs. LoRA fine-tuning runs 1.25x faster than full fine-tuning under expert parallelism. It targets B200/B300 GPUs but steps down to older hardware.
What changed
Distributed training required Megatron or stock Hugging Face tooling, with porting and new checkpoint formats.
What it unlocks
Running distributed training on Hugging Face models without leaving the Hugging Face ecosystem or rewriting model code.
- 2.3–2.8x throughput vs stock TRL
- ~100 lines to add a model family
- 1.25x faster LoRA vs full fine-tuning
- 9,600–9,900 tok/s/GPU on gpt-oss-20b
What you need to act on it
- Hugging Face model family
- GPU cluster with NVLink or similar interconnect
Sources