Michael Noukhovitch published a paper and blog post on the Matthew Effect in RL for LLMs. Easy problems improve far more than hard ones during training. The effect appears in math, code and agentic RL benchmarks. He proposes Never Give Up, an adaptive sampling method. NGU retries hard problems with more samples while filtering easy ones quickly. It outperforms fixed k-values on GSM8k and DeepScaler math tasks. NGU also improves code RL on Manufactoria by solving harder tests. The method addresses signal efficiency, not just signal loss. Stale completions are a challenge, mitigated by age thresholds and baseline rescaling. The Matthew Effect resembles primacy bias but has different causes.
What changed
RL post-training improved easy problems far more than hard ones, leaving hard problems unsolved.
What it unlocks
Adaptive sampling in RL for LLMs can now focus compute on hard problems without wasting it on easy ones.
- 0% to 3.8% to 22.7% initial pass@1 for hard/medium/easy AIME problems
- k=4 to k=32 completions compared
- p=0.9 NGU probability
- ~120 H100 hours training time for DeepScaler
What you need to act on it
- asynchronous RL for LLMs setup
- GRPO or similar RL method
Sources