mlsanityData to 14 Oct 2026

Mind the Gap: Benchmarking LLM Uncertainty and Calibration with Specialty-Aware Clinical QA and Reasoning-Based Behavioural Features

arXiv 2506.10769 ↗First posted 12 Jun 2025cs.CLcited by 1

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

Reliable uncertainty quantification (UQ) is essential when employing large language models (LLMs) in high-risk domains such as clinical question answering (QA). In this work, we evaluate uncertainty estimation methods for clinical QA focusing, for the first time, on eleven clinical specialties and six question types, and across ten open-source LLMs (general-purpose, biomedical, and reasoning models), alongside representative proprietary models. We analyze score-based UQ methods, present a case study introducing a novel lightweight method based on behavioral features derived from reasoning-oriented models, and examine conformal prediction as a complementary set-based approach. Our findings reveal that uncertainty reliability is not a monolithic property, but one that depends on clinical specialty and question type due to shifts in calibration and discrimination. Our results highlight the need to select or ensemble models based on their distinct, complementary strengths and clinical use.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 8 of its 53 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

8 of 53 references.

References · 8

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 274
  3. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  4. 2024Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Datacited by 21
  5. 2024Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measurecited by 18
  6. 2024Hippocrates: An Open-Source Framework for Advancing Large Language Models in Healthcarecited by 15
  7. 2025The challenge of uncertainty quantification of large language models in medicinecited by 14
  8. 2024Uncertainty Estimation of Large Language Models in Medical Question Answeringcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Unanimously Wrong: Certified Abstention from How Medical LLM Consensus Formscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Semantic Calibration Prevails Where Token Confidence Fails: Benchmarking Long-Form Scientific QA
  2. 2025Tools in the Loop: Quantifying Uncertainty of LLM Question Answering Systems That Use Tools
  3. 2026Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure?
  4. 2026Mind the Ambiguity: Aleatoric Uncertainty Quantification in LLMs for Safe Medical Question Answering
  5. 2024MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertainty
  6. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey
  7. 2024Enhancing Healthcare LLM Trust with Atypical Presentations Recalibration
  8. 2026Knowing When to Abstain: Medical LLMs Under Clinical Uncertainty