mlsanityData to 7 Oct 2026

Probabilistic Medical Predictions of Large Language Models

arXiv 2408.11316 ↗First posted 21 Aug 2024cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have shown promise in clinical applications through prompt engineering, allowing flexible clinical predictions. However, they struggle to produce reliable prediction probabilities, which are crucial for transparency and decision-making. While explicit prompts can lead LLMs to generate probability estimates, their numerical reasoning limitations raise concerns about reliability. We compared explicit probabilities from text generation to implicit probabilities derived from the likelihood of predicting the correct label token. Across six advanced open-source LLMs and five medical datasets, explicit probabilities consistently underperformed implicit probabilities in discrimination, precision, and recall. This discrepancy is more pronounced with smaller LLMs and imbalanced datasets, highlighting the need for cautious interpretation, improved probability estimation methods, and further research for clinical use of LLMs.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 26 of its 50 references that are in the map. First posted on arXiv 21 Aug 2024.

Builds on

26 of 50 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  7. 2024Qwen2 Technical Reportcited by 2,076
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  11. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  12. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
14 more
  1. 2020Aligning AI With Shared Human Valuescited by 706
  2. 2024Yi: Open Foundation Models by 01.AIcited by 706
  3. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 165
  4. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  5. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 102
  6. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  7. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  8. 2023A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Taskscited by 36
  9. 2024NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoningcited by 31
  10. 2024Guiding Clinical Reasoning with Large Language Models via Knowledge Seedscited by 19
  11. 2022Assessing the communication gap between AI models and healthcare professionals: explainability, utility and trust in AI-driven clinical decision-makingcited by 14
  12. 2023Large Language Models Leverage External Knowledge to Extend Clinical Insight Beyond Language Boundariescited by 10
  13. 2024Understanding Token Probability Encoding in Output Embeddingscited by 6
  14. 2024Using Pretrained Large Language Model with Prompt Engineering to Answer Biomedical Questionscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 7.

16 papers.

Sort:

By area:

  1. 2025BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Textcited by 11
  2. 2025AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretationcited by 8
  3. 2025MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modellingcited by 7
  4. 2025MedKGent: A Large Language Model Agent Framework for Constructing Temporally Evolving Medical Knowledge Graphcited by 5
  5. 2025LLMs in Disease Diagnosis: A Comparative Study of DeepSeek-R1 and O3 Mini Across Chronic Health Conditionscited by 3
  6. 2025Rehabilitation Exercise Quality Assessment and Feedback Generation Using Large Language Models with Prompt Engineeringcited by 3
  7. 2025Scalable Medication Extraction and Discontinuation Identification from Electronic Health Records Using Large Language Modelscited by 1
  8. 2025Multi-modal Integration Analysis of Alzheimer's Disease Using Large Language Models and Knowledge Graphscited by 0
  9. 2025HySemRAG: A Hybrid Semantic Retrieval-Augmented Generation Framework for Automated Literature Synthesis and Methodological Gap Analysiscited by 0
  10. 2026Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultationcited by 0
  11. 2026EpiScreen: Early Epilepsy Detection from Electronic Health Records with Large Language Modelscited by 0
  12. 2026CLSGen: A Dual-Head Fine-Tuning Framework for Joint Probabilistic Classification and Verbalized Explanationcited by 0
4 more
  1. 2026Language-Induced Priors for Domain Adaptationcited by 0
  2. 2026TRIAGE: Dialectical Reasoning for Explainable Risk Prediction on Irregularly Sampled Medical Time Series with LLMscited by 0
  3. 2026Evidence-Grounded AI for Musculoskeletal Carecited by 0
  4. 2026Large Language Models Show Metacognitive Sensitivity in Medical Reasoningcited by 0