# Mind the Gap: Benchmarking LLM Uncertainty and Calibration with Specialty-Aware Clinical QA and Reasoning-Based Behavioural Features

arXiv: https://arxiv.org/abs/2506.10769 · HTML: https://mlsanity.com/paper/2506.10769/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 8 of its 53 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 8 of 8)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://mlsanity.com/paper/2311.16079/) (2023)
- [Uncertainty in Natural Language Generation: From Theory to Applications](https://mlsanity.com/paper/2307.15703/) (2023)
- [Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data](https://mlsanity.com/paper/2408.13833/) (2024)
- [Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measure](https://mlsanity.com/paper/2412.15176/) (2024)
- [Hippocrates: An Open-Source Framework for Advancing Large Language Models in Healthcare](https://mlsanity.com/paper/2404.16621/) (2024)
- [The challenge of uncertainty quantification of large language models in medicine](https://mlsanity.com/paper/2504.05278/) (2025)
- [Uncertainty Estimation of Large Language Models in Medical Question Answering](https://mlsanity.com/paper/2407.08662/) (2024)

## Built on by (most cited first, 1 of 1)

- [Unanimously Wrong: Certified Abstention from How Medical LLM Consensus Forms](https://mlsanity.com/paper/2610.07570/) (2026)

## Abstract (arXiv)

Reliable uncertainty quantification (UQ) is essential when employing large language models (LLMs) in high-risk domains such as clinical question answering (QA). In this work, we evaluate uncertainty estimation methods for clinical QA focusing, for the first time, on eleven clinical specialties and six question types, and across ten open-source LLMs (general-purpose, biomedical, and reasoning models), alongside representative proprietary models. We analyze score-based UQ methods, present a case study introducing a novel lightweight method based on behavioral features derived from reasoning-oriented models, and examine conformal prediction as a complementary set-based approach. Our findings reveal that uncertainty reliability is not a monolithic property, but one that depends on clinical specialty and question type due to shifts in calibration and discrimination. Our results highlight the need to select or ensemble models based on their distinct, complementary strengths and clinical use.