# CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR

arXiv: https://arxiv.org/abs/2608.07796 · HTML: https://mlsanity.com/paper/2608.07796/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 10 of its 42 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 10 of 10)

- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Agents' Last Exam](https://mlsanity.com/paper/2606.05405/) (2026)
- [PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments](https://mlsanity.com/paper/2605.02240/) (2026)
- [A Framework for Formalizing LLM Agent Security](https://mlsanity.com/paper/2603.19469/) (2026)
- [Knowing When to Abstain: Medical LLMs Under Clinical Uncertainty](https://mlsanity.com/paper/2601.12471/) (2026)
- [LongMedBench: Benchmarking Medical Agents for Long-Horizon Clinical Decision-Making](https://mlsanity.com/paper/2607.09322/) (2026)
- [EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning](https://mlsanity.com/paper/2606.23301/) (2026)
- [CliCARE: Grounding Large Language Models in Clinical Guidelines for Decision Support over Longitudinal Cancer Electronic Health Records](https://mlsanity.com/paper/2507.22533/) (2025)
- [ClinEnv: An Interactive Multi-Stage Long Horizon EHR Environment for Agents](https://mlsanity.com/paper/2606.02568/) (2026)
- [ClinDet-Bench: Beyond Abstention, Evaluating Judgment Determinability of LLMs in Clinical Decision-Making](https://mlsanity.com/paper/2602.22771/) (2026)

## Built on by (most cited first, 2 of 2)

- [READY or Not: Reliable Enterprise Agent Deployment](https://mlsanity.com/paper/2609.02095/) (2026)
- [Constructing and Evaluating Clinical Reasoning Trajectories for Medical Agent](https://mlsanity.com/paper/2609.05090/) (2026)

## Abstract (arXiv)

Large language models perform strongly on medical knowledge benchmarks, but reliable clinical deployment requires agents to conduct defensible investigations over heterogeneous, longitudinal records: determining what evidence is needed, retrieving and reconciling structured and free-text data, grounding conclusions in verifiable evidence, and deferring cases that cannot be resolved reliably. We introduce CliniCARE-Bench (Clinical Calibrated Audit of Medical Reasoning in EHR), a benchmark for retrospective clinical audit: 25 clinician-validated scenarios instantiated as 750 patient-specific cases over real-patient-derived MIMIC-IV data. Systems investigate each case through a governed, logged tool environment for record retrieval, computation, and policy access, and return one of four verdicts---Yes, No, Indeterminate: Lack of Data, or Indeterminate: Medically Ambiguous---the last two separating missing evidence from residual medical ambiguity. Beyond verdict accuracy, we score patient-evidence and policy grounding, process adherence, calibrated abstention, reliability, and efficiency against case-level reference verdicts produced by independent multi-model adjudication and calibrated against Clinical Board review. Every retrieval, computation, and report is replayable, so the investigation trace is inspectable and scorable. To our knowledge, CliniCARE-Bench is the first deployment-oriented clinical-agent benchmark to jointly evaluate real longitudinal EHR investigation, claim-level evidence grounding, governing-policy use, process adherence, and calibrated abstention within a common patient-level adjudication framework. Across 16 agentic systems, four-way accuracy spans 65.3-76.1%, but raw accuracy overstates investigation quality. Defect-free accuracy, which credits a verdict only when correct and free of prohibited shortcuts, is 4.8-14.8 points lower and reorders the leaderboard.