CyberGym-E2E is a benchmark testing AI agents on 920 real-world vulnerabilities. It covers the full defensive lifecycle: discover, prove and patch. Agents are evaluated across four cumulative validation stages. The benchmark uses 139 open-source projects. It places agents in realistic build environments. Success is measured by crash fixes and passing functionality tests. A gap exists between passing tests and fixing the intended vulnerability. Agents often find alternative vulnerabilities. Some patches are shallow and leave root causes untouched. Success rates rise with cost budgets before plateauing. Patch-only performance plateaus earlier than end-to-end discovery. Lifting cost caps shows sustained reasoning in top models. The benchmark scales with new vulnerabilities automatically.
What it unlocks
Measuring AI agents' full cybersecurity lifecycle performance from discovery to patching.
- 920 tasks
- 139 open-source projects
- 4 validation stages
- $10 cost cap for evaluation
Sources