NVIDIA researchers questioned whether on-policy distillation is always best. They proposed Semi-OPD, which uses offline student rollouts for distillation. Semi-OPD outperformed standard on-policy distillation in 14 of 17 tested pairs. It delivered up to 13.6% higher accuracy and 11.4 times faster training. The method works best when teacher and student models are misaligned. High context length can weaken the distillation signal in standard on-policy approaches. Semi-OPD remains stable by using shorter contexts while covering full trajectories. The work suggests rethinking when to use on-policy distillation. It also calls for stronger variants with better teacher-student alignment. The paper covers models from 1.5 billion to 235 billion parameters.
What changed
On-policy distillation was assumed beneficial for all teacher-student pairs.
What it unlocks
Using offline student rollouts to distill models with higher accuracy and faster training.
- 17 teacher-student pairs
- +13.6% accuracy
- 11.4x training speedup
- 1.5B to 235B parameters
What you need to act on it
- misaligned teacher-student pairs
- high context length
Sources