A 24-author study led by Peter Kirgis, with Sayash Kapoor, Rishi Bommasani, Helen Toner and Arvind Narayanan among the co-authors, introduced "shadow evaluations" as a method for measuring progress toward automated AI research. An agent is given the central open-ended research question of a high-quality unpublished paper, and the paper's original authors grade the resulting output. The team ran the method on two unpublished NeurIPS 2026 submissions, allowing frontier agents six days and thousands of dollars of compute. The agents completed all engineering work without human help but made no substantial progress on the research questions, and both outputs were unambiguously rejected by the original authors. Five recurring failure modes were identified: poor judgment about the publishable-research bar, uncreative responses to research design shortcomings, ineffective backtracking from dead ends, poor resource awareness, and instruction drift.

• arxiv.org2026-08-03