Physics faculty and graduate researchers re-graded frontier model answers on six widely used physics benchmarks. They found most answers marked wrong were not reasoning failures. Instead the fault lay with automated graders, wrong reference solutions and badly specified questions. The experts corrected the reference answers and repaired or removed flawed problems. Scores then rose sharply for GPT-5.6-Sol on HLE-Physics and CMT-Benchmark. Results also improved on audited parts of UGPhysics, PRISM-Physics and PHYBench. The authors conclude current benchmarks understate model ability on well-posed physics problems. They say these closed-answer tests are near saturation and harder expert-checked evaluations are needed. The work covered text-only problems with checkable final answers. Fifty-one authors from Yale and elsewhere contributed.
What changed
Published physics benchmark scores suggested frontier models still failed advanced physics problems.
What it unlocks
Judging model physics ability from expert-corrected test sets rather than raw benchmark scores.
- HLE-Physics: 47.3% to 78.7%
- CMT-Benchmark: 61.0% to 87.2%
- 94.4% pass@4 on 54 CritPt items
- 6 benchmarks audited
Sources