Research2026-08-28
Epoch AI published a benchmark that tests whether AI systems get better at an unfamiliar task when allowed to practise. Models repeatedly play a little-known campaign board game, keeping written notes between attempts, and are scored on their final two rounds. Early results show scores staying roughly flat across repeated play and well below skilled human players, who reached the maximum score.
What changed
Most AI benchmarks score a single attempt rather than measuring improvement across repeated tries.
What it unlocks
Comparing how much frontier models improve at an unfamiliar long task when given practice runs and notes.
- 21 objectives available per playthrough
- top human scored 21/21
- 13 human baseline participants
- 2-4 hours per human playthrough
Sources