Reinforcement learning for large language models often fails when training and inference engines differ. The failure comes from drift that builds up with every training step. The authors propose an additive score centering correction to cancel this drift. Score centering alone matches or beats importance sampling under quantization. The gap widens as the mismatch gets worse. Score centering also combines with importance sampling. Their combined approach outperforms pure importance-sampling baselines in staleness tests.
What changed
Off-policy reinforcement learning was unstable under training-inference mismatch due to accumulating drift.
What it unlocks
Stabilizing reinforcement learning for large language models under training-inference mismatch.
- 0.6B to 30B parameters
- gap grows with severity of mismatch
Sources