Vercel published DeepsecBench, a public leaderboard measuring how well AI models find security flaws in application code. Models are scored on how many known flaws they catch, how often they raise false alarms, and what each scan costs and takes in time. Even the best model found under a third of the known flaws, and the underlying test code is kept secret so models cannot be trained on it.
What changed
Vercel previously said thorough scanning of production code required the most capable and most expensive models, with no published comparison of accuracy against cost.
What it unlocks
Choosing which model to run for each type of security scan by comparing measured detection rates, false-positive rates, cost and runtime.
- top score 35.58 (GPT-5.6 Sol, xhigh), 30.7% recall, $55.98
- Kimi K3 scored 17.56 for $12.38; Grok 4.5 scored 15.58 for $5.60
- 231 human-judged findings across 50 entry-point files
- a full production codebase pass estimated at roughly $1,200 (Kimi K3) to over $5,000 (top OpenAI model)
What you need to act on it
- use of the deepsec scanning tool
- an AI Gateway API key or a linked Vercel project
Sources