Princeton, Stanford, UK AI Security Institute and others published a survey on AI agent evaluation. It identifies 33 shifts across methodology, research scope and evaluator ecosystem. The paper argues existing model benchmarks fail for agents. It proposes new metrics beyond accuracy for agent performance. It calls for statistical rigor to handle long, stochastic agent trajectories. It highlights contamination risks from tools and environments. It recommends dynamic benchmarks to address saturation. It stresses the need for LLM judges and human oversight. It notes rising costs and infrastructure demands for agent evaluation. It explores field experiments and real-world usage data. It discusses evaluator independence and funding impacts. It outlines best practices for each trend.
Sources