Researchers studied when larger batch sizes speed up reinforcement learning for large language models. They separated algorithmic and systems effects to compare learning and execution. Algorithmic tests showed similar learning curves across batch sizes with retuned hyperparameters. Systems tests found larger batches improve generation throughput by up to 2.29x on fixed hardware. Combining higher throughput with learning-rate retuning cut time-to-target by up to 29%. Without retuning, larger batches were slower despite higher throughput. The rule is that larger batches help only when throughput gain beats samples-to-target penalty. Experiments used GRPO and PPO methods to support the findings.

• 2.29x generation throughput gain
• 29% time-to-target reduction
• 16 pages, 9 figures