Researchers introduced the LIFT architecture for language models. It enables deep-layer representations to feed back to shallower layers. The method uses teacher supervision with precomputed states from an existing model. Models are trained to predict both the next token and the next state. Inference uses the model’s own predicted states with minimal overhead. Experiments show LIFT outperforms standard Transformers on language modeling and reasoning tasks. It also excels in procedural tasks under token-matched budgets. A tiny LIFT model beat same-size Transformers trained on eight times more data. The paper demonstrates scalable teacher supervision for state propagation.
What changed
Transformer language models used only feed-forward paths and token-based information flow.
What it unlocks
Language models can now propagate state across generation steps using deep-to-shallow feedback.
- 135M to 1B parameters
- 8x more data
- 286 KB
What you need to act on it
- teacher-forced pretraining with precomputed states
- additional parameters added to the model
Sources