CliniCARE-Bench is a clinician-validated benchmark for evaluating AI agents on retrospective clinical audit tasks using real longitudinal EHR data from MIMIC-IV. It comprises 25 expert-authored scenarios instantiated across 750 patient cases, requiring agents to retrieve evidence, apply governing clinical policies, and return one of four verdicts (Yes, No, Indeterminate: Lack of Data, or Indeterminate: Medically Ambiguous). Beyond accuracy, the benchmark scores evidence grounding, policy citation, process adherence, and calibrated abstention. Testing 16 frontier agentic systems reveals that raw accuracy (65.3–76.1%) masks significant process defects: "defect-free" accuracy drops 4.8–14.8 points when prohibited reasoning shortcuts are penalized, and every system systematically under-abstains on cases requiring deferral.
CliniCARE-Bench is a clinician-validated benchmark for evaluating AI agents on retrospective clinical audit, one of the most demanding real-world tasks for clinical AI. Rather than testing static medical knowledge, it places agents in a governed, fully logged tool environment built on real, de-identified patient data from MIMIC-IV, where they must investigate a case much like a human auditor would: deciding what evidence is needed, retrieving it across structured records and free-text notes, reconciling conflicting sources, applying the relevant clinical guideline or policy, and citing the evidence behind every claim.
The benchmark comprises 25 scenarios developed and validated by a Clinical Board, spanning 14 medical specialties and ten distinct reasoning skills, from clinical score computation to negative reasoning and longitudinal synthesis across years of records. These are instantiated into 750 patient-specific cases, each requiring one of four verdicts: Yes, No, Indeterminate: Lack of Data, or Indeterminate: Medically Ambiguous. Separating these two indeterminate classes is central to the benchmark's design; it distinguishes cases where more evidence could resolve the question from cases that remain genuinely ambiguous even after a complete review, so that principled abstention is measured and rewarded rather than treated as a shortcut.
Every retrieval, computation, and citation an agent makes is logged and replayable, allowing the benchmark to score not just whether an agent reached the right answer but whether its investigation was defensible. This includes patient-evidence grounding, policy grounding, and adherence to a weighted rubric of required actions and prohibited shortcuts.
In an evaluation of 16 leading agentic systems built on frontier models, accuracy tops out at 76.1%, and correct answers frequently mask flawed investigations: "defect-free" accuracy, which credits a verdict only when the process behind it is also sound, falls by 4.8 to 14.8 percentage points and reorders the leaderboard entirely. Every system evaluated also under-abstains, more readily committing to a definitive answer than deferring on cases the record cannot actually settle. These findings point to a meaningful gap between how clinical AI performs on paper and what would be required for trustworthy, auditable deployment in real clinical workflows.