8090 Labs ran Datacurve’s DeepSWE v1.1 on seven models across five harnesses. The harness alone changed a model’s success rate by up to 35 points. GLM 5.3 Flash scored 24% in Codex but 59% in DeepSeek’s harness. GPT-5.6 Sol varied by only 6 points across harnesses. No model performed best in its maker’s own harness. Claude Fable 5.1 scored 41% in Claude Code but 61% in Codex. Reliability varied widely between harnesses for the same model. Cost per success differed sharply from cost per run. Language-specific performance gaps appeared between harnesses. Teams are advised to test model and harness together. They should also check repeatability and cost per success.
What changed
Coding benchmarks ranked models without accounting for the harness used.
What it unlocks
Teams can now evaluate how much a harness affects a model's performance, reliability, and cost.
- 35-point swing in model success from harness choice
- 113 real coding tasks benchmarked
- 2 runs per task per model-harness pair
- 9-point gap or less considered negligible
Sources