Specific released Real-SWE, a benchmark built from private company codebases. The firm licensed the code from real companies and used tasks their engineers actually worked on. It tested eight model-and-harness pairings across ten tasks, scoring 640 runs. Anthropic's Fable 5.1 in Claude Code led on the share of tasks solved. OpenAI's GPT-6 Astra in Codex CLI and Google's Gemini 3.8 Flash in Gemini CLI followed. Most tasks were solved rarely, and one was never solved at all. Tasks touched billing, tax rules, customer migrations and cloud infrastructure. Specific says models struggle with company-specific coding conventions and often skip requirements. Longer runs failed about as often as short ones. The sample tasks are available only on request.
What changed
Coding benchmarks used public or synthetic tasks rather than licensed private code.
What it unlocks
Comparing coding agents on private enterprise tasks they cannot have seen in training.
- 38.8% top resolution rate
- 640 scored rollouts
- 6 of 10 tasks below 15%
- 11 files edited, median
What you need to act on it
- request access to view the sample tasks
Sources