IBM Research introduced a new metric called Pass^k to measure agent consistency. It found a 24.4-point gap between average success and consistent success for a ReAct agent. The gap arises from flat decision distributions that flip under small perturbations. IBM built a Consistency Analyzer to detect flip-prone steps in agent trajectories. It generates consistency guidelines that reduce the gap from 24.4pp to 12.0pp. The guidelines improve Pass^5 by 16.0pp on same tasks and 13.0pp on similar tasks. Mean accuracy remains unchanged. The method works offline with one recorded trajectory per task. It requires no ground truth or model internals. Full details are in an arXiv technical report.
What changed
Agents were evaluated only on average success rates, hiding variability across runs.
What it unlocks
Measuring and reducing the gap between average success and consistent success for agent tasks.
- 77.4% Mean@5 success rate
- 53.0% Pass^5 success rate
- 24.4-point consistency gap
- 12.0-point gap after guidelines
What you need to act on it
- a recorded agent trajectory
- access to ALTK-Evolve framework
- GPT-4.1 or equivalent model
Sources