Researchers showed architectural changes can alter scaling exponents in model training. Looped transformers provide a mechanism for model growth by increasing loops. Model growth with shared weights yields the largest exponent changes. A 7.4B model matched GPT-3 13B on CORE with roughly 20 times less compute. Boundary operators in vanilla transformers also increase exponents but less so. Looping acts as a regularizer in data-constrained multi-epoch training. Increasing loops with scale is compute-optimal in this setting. The work frames gains as increasing usable transformer depth for a fixed budget. Efficiency gains grow with model scale. Code is available on GitHub.

• 7.4B model matches GPT-3 13B
• $20× less compute
• 44 pages