Release2026-10-23

Microsoft released SDBench, a benchmark for sequential medical diagnosis. It tests AI systems that gather information, order tests, and commit diagnoses step by step. A gatekeeper returns real or synthetic test results, and a judge scores diagnoses on a 1–5 scale. The benchmark ships with a small synthetic dataset for development. Clinically sourced datasets require separate access. Configurations pin gatekeeper and judge models for comparability. Costing modules estimate monetary test costs. The Science paper configuration reproduces published baselines. A website hosts full documentation. Local, Azure, and OpenRouter model routing are supported.

What it unlocks

Evaluating AI systems for sequential medical diagnosis with standardized cases and scoring.

What you need to act on it

  • Python environment with uv
  • API keys for model providers
  • Access to clinical datasets for non-synthetic evaluation

Sources

Send this to someone who needs it

Shares the story and its sources. Nothing about you.

What does this mean for your job?

This is the story as everyone gets it. Once a week we send you the version written for your role — what changed, why it matters for the work you actually do, and one thing to try. Free while we tune it.