PhysioNet released MIMIC-IV-Ext-SDBench, a dataset of 488 hospital cases for AI diagnostic testing. Each case includes a brief vignette, discharge summary, and ground truth diagnosis. 71 test cases were reviewed by clinicians. The dataset supports the Sequential Diagnosis Benchmark for iterative reasoning. It splits into 163 training, 254 validation, and 71 test cases. A baseline agent achieved 76.6% accuracy on 325 cases. A Diagnostic Orchestrator agent reached 80.3% without budget limits. Mean workup costs were about $4,240 and $4,980 per case. Cases were curated from MIMIC-IV emergency department admissions. Large language models aided data preparation under PhysioNet guidance.
What it unlocks
Evaluating AI agents on iterative diagnostic reasoning with real hospital cases.
- 488 case records
- 71 clinician-reviewed test cases
- 76.6% diagnostic accuracy on 325 cases
- 80.3% accuracy for DxO agent
What you need to act on it
- access to MIMIC-IV data
- SDBench evaluation harness
Sources