mlsanityData to 7 Oct 2026

Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultation

arXiv 2601.15645 ↗First posted 22 Jan 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large-scale language models (LLMs) often offer clinical judgments based on incomplete information, increasing the risk of misdiagnosis. Existing studies have primarily evaluated confidence in single-turn, static settings, overlooking the coupling between confidence and correctness as clinical evidence accumulates during real consultations, which limits their support for reliable decision-making. We propose the first benchmark for assessing confidence in multi-turn interaction during realistic medical consultations. Our benchmark unifies three types of medical data for open-ended diagnostic generation and introduces an information sufficiency gradient to characterize the confidence-correctness dynamics as evidence increases. We implement and compare 27 representative methods on this benchmark; two key insights emerge: (1) medical data amplifies the inherent limitations of token-level and consistency-level confidence methods, and (2) medical reasoning must be evaluated for both diagnostic accuracy and information completeness. Based on these insights, we present MedConf, an evidence-grounded linguistic self-assessment framework that constructs symptom profiles via retrieval-augmented generation, aligns patient information with supporting, missing, and contradictory relations, and aggregates them into an interpretable confidence estimate through weighted integration. Across two LLMs and three medical datasets, MedConf consistently outperforms state-of-the-art methods on both AUROC and Pearson correlation coefficient metrics, maintaining stable performance under conditions of information insufficiency and multimorbidity. These results demonstrate that information adequacy is a key determinant of credible medical confidence modeling, providing a new pathway toward building more reliable and interpretable large medical models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 66 references that are in the map. First posted on arXiv 22 Jan 2026.

Builds on

36 of 66 references.

References · 36

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  5. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  6. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  7. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  9. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  10. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  11. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  12. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 193
24 more
  1. 2019Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentationcited by 162
  2. 2024Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantificationcited by 159
  3. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  4. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
  5. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  6. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  7. 2022DDXPlus: A New Dataset For Automatic Medical Diagnosiscited by 87
  8. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 67
  9. 2024SafeBench: A Safety Evaluation Framework for Multimodal Large Language Modelscited by 63
  10. 2023A Review of Uncertainty Estimation and its Application in Medical Imagingcited by 59
  11. 2023Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Modelscited by 40
  12. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructionscited by 33
  13. 2022Rainproof: An Umbrella To Shield Text Generators From Out-Of-Distribution Datacited by 21
  14. 2024MediTOD: An English Dialogue Dataset for Medical History Taking with Comprehensive Annotationscited by 21
  15. 2024Probabilistic Medical Predictions of Large Language Modelscited by 16
  16. 2025Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generationcited by 15
  17. 2025The challenge of uncertainty quantification of large language models in medicinecited by 14
  18. 2025SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agentscited by 14
  19. 2024Uncertainty Estimation of Large Language Models in Medical Question Answeringcited by 10
  20. 2024NoVo: Norm Voting off Hallucinations with Attention Heads in Large Language Modelscited by 9
  21. 2024Sparse Bayesian Networks: Efficient Uncertainty Quantification in Medical Image Analysiscited by 8
  22. 2024Enhancing Healthcare LLM Trust with Atypical Presentations Recalibrationcited by 6
  23. 2024Uncertainty Quantification for Clinical Outcome Predictions with (Large) Language Modelscited by 3
  24. 2025RoboView-Bias: Benchmarking Visual Bias in Embodied Agents for Robotic Manipulationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Medical Reasoning with Large Language Models: A Survey and MR-Bench
  2. 2025Automating Expert-Level Medical Reasoning Evaluation of Large Language Models
  3. 2026CLEAR: Revealing How Noise and Ambiguity Degrade Reliability in LLMs for Medicine
  4. 2025Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases
  5. 2026Measuring Epistemic Resilience of LLMs Under Misleading Medical Context
  6. 2025Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models
  7. 2026MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Models
  8. 2025MediEval: A Unified Medical Benchmark for Patient-Contextual and Knowledge-Grounded Reasoning in LLMs