Researchers introduced Information-Proximal Self-Distillation Fine-Tuning (iSDFT) for continual learning in LLMs. iSDFT treats the teacher as a budgeted information source at each token. It selects the closest distribution to the student that meets a teacher-information constraint. A frozen base policy anchor prevents cumulative drift from the original model. iSDFT improved vanilla SDFT in 7 of 8 model–task settings and matched it in the remaining one. It kept 73% of evaluations within 0.5 points of the base model versus 52% for the strongest baseline. iSDFT achieved the largest mean improvement on all ten additional mathematics and coding benchmarks. The method balances local plasticity toward new tasks with long-term stability of existing capabilities.
What changed
On-policy self-distillation fine-tuning always distilled toward the full demonstration-conditioned teacher with no control over teacher influence.
What it unlocks
Controlling how much and when teacher information is introduced during continual learning while preserving broader model capabilities.
- 7 of 8 model–task settings improved
- 73% within 0.5 points of base model
- 52% for strongest baseline
- 10 additional benchmarks evaluated
Sources