Researchers at the University of Illinois, Bridgewater and Thinking Machines fine-tuned a single open model to turn plain-language business questions into database queries at accuracy matching human experts, without the multi-step pipelines other systems use. The gains came from cleaning up mislabelled training data and changing how correct answers are scored during training. Costs per query were far below those of leading commercial models, and the data, code and recipe were published.
What changed
The best text-to-SQL systems relied on multi-step pipelines and trailed the human accuracy mark by around 11 points.
What it unlocks
Training a single model to turn business questions into database queries at human-level accuracy without a multi-step pipeline, using the published data, code and recipe.
- 92.97% vs 92.96% human level
- $0.56 per task with 16 samples
- $0.035 per task, single sample
- 61.1% of audited training items flawed
What you need to act on it
- access to Tinker fine-tuning
- expert-verified training data
- compute for reinforcement learning
Sources