Microsoft released SDBench, a benchmark for sequential diagnosis in medicine. It scores AI systems on diagnostic accuracy and test cost. Each case runs a loop between a diagnostic agent, a gatekeeper and a judge. The agent orders tests, asks questions or commits a diagnosis turn by turn. The judge scores the final diagnosis on a 1–5 scale. Cases come from NEJM, MIMIC or a bundled synthetic set. NEJM and MIMIC require separate access and licensing. The synthetic set is open and included for quick evaluation. The harness runs with uv and supports swapping models or datasets. A coding agent can automate setup and execution.
What it unlocks
Benchmarking AI systems that iteratively order tests, ask questions, and commit diagnoses in medical cases.
What you need to act on it
- access to NEJM or MIMIC datasets for clinical cases
- OpenRouter API key for model inference
Sources