Researchers studied efficient recurring evaluation for a production LLM agent. They used 574 historical benchmark runs split into calibration and held-out periods. They compared random sampling, historical caching, fixed subsets, and IRT-based adaptive testing. Multidimensional 2PL adaptive testing achieved the best score fidelity. It executed 200 questions, 38.5% of a full run, with 1.03 pp MAE. They deployed difficulty-stratified fixed subsets for operational simplicity. These subsets transferred without recalibration to five other agent families. They remained stable across calibration windows as short as one day. The paper reports practical recommendations for recurring production-agent evaluation.
What it unlocks
Running recurring production-agent evaluation with fewer questions while maintaining score fidelity.
- 574 historical benchmark runs
- 200 questions for adaptive testing
- 38.5% of a full run
- 1.03 pp MAE
Sources