Trustworthy Agents for Electronic Health Records through Confidence Estimation
On the map: LLM agents and tool use → Medical and scientific domain agents
Abstract (arXiv)
Large language models (LLMs) show promise for extracting information from Electronic Health Records (EHR) and supporting clinical decisions. However, deployment in clinical settings faces challenges due to hallucination risks. We propose Hallucination Controlled Accuracy at k% (HCAcc@k%), a novel metric quantifying the accuracy-reliability trade-off at varying confidence thresholds. We introduce TrustEHRAgent, a confidence-aware agent incorporating stepwise confidence estimation for clinical question answering. Experiments on MIMIC-III and eICU datasets show TrustEHRAgent outperforms baselines under strict reliability constraints, achieving improvements of 44.23%p and 25.34%p at HCAcc@70% while baseline methods fail at these thresholds. These results highlight limitations of traditional accuracy metrics in evaluating healthcare AI agents. Our work contributes to developing trustworthy clinical agents that deliver accurate information or transparently express uncertainty when confidence is low.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 16 of its 27 references that are in the map. First posted on arXiv 26 Aug 2025.
Builds on
16 of 27 references.
References · 16
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
- 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
- 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
- 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 122
- 2024On Verbalized Confidence Scores for LLMscited by 83
- 2019Text-to-SQL Generation for Question Answering on Electronic Medical Recordscited by 77
- 2024Factual Confidence of LLMs: on Reliability and Robustness of Current Estimatorscited by 61
4 more
- 2023Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AIcited by 33
- 2024MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Modelscited by 21
- 2023Assessing the Reliability of Large Language Model Knowledgecited by 13
- 2025MedHal: a Synthetic Dataset for Medical Hallucination Detectioncited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Trust but Verify: Mitigating Medical Hallucinations via Post-Hoc Adversarial Auditing and Multi-Agent Feedback Loops
- 2026Trajectory-Derived Confidence for Reliable, Resource-Aware Clinical Text-to-SQL Agents
- 2025Trustworthy Medical Question Answering: An Evaluation-Centric Survey
- 2026CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs
- 2026MedAgent-R1: Faithfulness-Aware Reinforcement Learning for Evidence-Grounded Medical Reasoning
- 2025MedMMV: A Controllable Multimodal Multi-Agent Framework for Reliable and Verifiable Clinical Reasoning
- 2026CuraView: A Multi-Agent Framework for Medical Hallucination Detection with GraphRAG-Enhanced Knowledge Verification
- 2026ART: Action-based Reasoning Task Benchmarking for Medical AI Agents