Up to 40% of patients on immune checkpoint inhibitors develop immune-related adverse events. Manual chart review is slow and inconsistent for detecting them. Researchers built an agentic AI system to extract irAE details from clinical notes. It achieved an F1 score of 0.92 for detecting current events in tests. It scored 0.66 for grading severity across multiple classes. Self-consistency checks raised the F1 score by 0.14. The system cost about two cents per note to run. In a three-month silent trial, its binary F1 ranged from 0.72 to 0.79. A crossover study showed it cut annotation time by 2.71 minutes. It also improved complete-match accuracy and interannotator agreement. The work targets six irAE types.
What changed
Manual chart review was the only method for irAE detection and grading.
What it unlocks
Faster, more consistent and accurate irAE assessment in clinical notes with human oversight.
- 40% of patients affected by irAEs
- macro-averaged F1 of 0.92 for event detection
- 0.66 F1 for severity grading
- $0.02 per note cost
What you need to act on it
- clinical notes in unstructured text
- human verification step
Sources