SphereLab studied when recurrence helps in looped language models. It found recurrence improves reasoning beyond training depth but harms knowledge tasks. Performance depends on how layers and loops are allocated. Non-recurrent output layers boost robustness to under-unrolling. Input and output layer placement varies with inference budget. Conventional initial-state injection offers limited robustness to varying depth. Researchers proposed history-state injection as an alternative. Channel-wise history-state injection with timestep conditioning preserves knowledge under extended unrolling. The design improves robustness across inference budgets. Results clarify when recurrent computation helps and where it fails.
What it unlocks
Scaling test-time computation in looped language models beyond training horizon while preserving knowledge robustness.
- 20 effective training depth
- 20 tokens per parameter training budget
- 22,573 surveyed, ±0.9 points
What you need to act on it
- pre-trained looped language model
- controlled inference budget
Sources