Researchers propose Harness-Delta Attribution to break down gains from harness evolution. It separates overfitting, test-time scaling, and residual general improvement. They tested it on 11 executor models and three frameworks across math, coding, creativity, and agentic tasks. In 21 settings with significant search-set gains, overfitting explained 54% of the gain on average. Test-time scaling explained 14% on average. Their combined share exceeded 50% in 14 of those settings. Residual and held-out transfer varied widely across tasks, models, and frameworks. Search-set gains often failed to transfer to held-out tasks. Validation signals were insufficient to prevent overfitting. The paper calls for attribution-aware evaluation in future research.

• 21 settings with significant search-set gains
• 54% of gain from overfitting on average
• 14% of gain from test-time scaling on average
• 14 settings where overfitting + scaling >50% of gain