Researchers found that training a weaker AI model on a stronger one's full worked examples can make it worse. They tested seven business agent tasks, using Qwen3-Coder and Gemma 4 as the weaker models. Each task first had its surrounding setup tuned automatically for the weaker model. That setup covers the instructions, tools and memory handling wrapped around a model. A stronger model then used that same setup more effectively than the weaker one. But copying the stronger model's whole approach cut performance on every task. The weaker model took on planning habits it could not carry out. The same copying did help when the setup had not been tuned first. The authors instead correct only the single step where the weaker model goes wrong. That keeps its own planning style and retains the gains from both changes.
What changed
Standard practice was to fine-tune a smaller model on a stronger model's complete task transcripts.
What it unlocks
Tuning a cheaper model without breaking the fit between it and its surrounding setup.
- 7 enterprise agent tasks
- 4 to 30 point drop on all 7 tasks
- 2 models tested: Qwen3-Coder, Gemma 4
What you need to act on it
- access to model weights for fine-tuning
- a stronger expert model to supervise
- a fine-tuning pipeline
Sources