Researchers found language models can transfer capabilities through unrelated text. They introduced Active Taskless Distillation to achieve this with single-word teacher responses. ATD uses near-tie prompts where a public ancestor model is indifferent between two words. The teacher’s choice on these prompts provides one-bit observations of its behavioral shadow. A student model trained on these pairs gains 5.34 pp on HumanEval+ over a matched control. Transfer also works for scientific knowledge, commonsense reasoning, and reading comprehension. The learned signal is source-specific and composable. Its strength tracks the teacher’s update strength. Code and details are available on GitHub.
What changed
Post-training was understood only through changes in target-task behavior.
What it unlocks
Transferring capabilities between language models using only single-word responses on unrelated prompts.
- 5.34 pp gain on HumanEval+
- 5,664 single-word teacher responses
- 128 words used in training
What you need to act on it
- a shared public ancestor model
- near-tie prompts where the ancestor is indifferent between two words
- access to the teacher model via a black-box interface
Sources