Position Paper On Diagnostic Uncertainty Estimation from Large Language Models: Next-Word Probability Is Not Pre-test Probability
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) are being explored for diagnostic decision support, yet their ability to estimate pre-test probabilities, vital for clinical decision-making, remains limited. This study evaluates two LLMs, Mistral-7B and Llama3-70B, using structured electronic health record data on three diagnosis tasks. We examined three current methods of extracting LLM probability estimations and revealed their limitations. We aim to highlight the need for improved techniques in LLM confidence estimation.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 7 of its 20 references that are in the map. First posted on arXiv 7 Nov 2024.
Builds on
7 of 20 references.
References · 7
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
- 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 100
- 2023Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicinecited by 83
- 2024When Raw Data Prevails: Are Large Language Model Embeddings Effective in Numerical Data Representation for Medical Machine Learning Applications?cited by 12
Built on by
Citing papers per year: 2025: 2.
2 papers.