mlsanityData to 14 Oct 2026

Evaluating Prompt Engineering Techniques for Accuracy and Confidence Elicitation in Medical LLMs

arXiv 2506.00072 ↗First posted 29 May 2025cs.CY cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper investigates how prompt engineering techniques impact both accuracy and confidence elicitation in Large Language Models (LLMs) applied to medical contexts. Using a stratified dataset of Persian board exam questions across multiple specialties, we evaluated five LLMs - GPT-4o, o3-mini, Llama-3.3-70b, Llama-3.1-8b, and DeepSeek-v3 - across 156 configurations. These configurations varied in temperature settings (0.3, 0.7, 1.0), prompt styles (Chain-of-Thought, Few-Shot, Emotional, Expert Mimicry), and confidence scales (1-10, 1-100). We used AUC-ROC, Brier Score, and Expected Calibration Error (ECE) to evaluate alignment between confidence and actual performance. Chain-of-Thought prompts improved accuracy but also led to overconfidence, highlighting the need for calibration. Emotional prompting further inflated confidence, risking poor decisions. Smaller models like Llama-3.1-8b underperformed across all metrics, while proprietary models showed higher accuracy but still lacked calibrated confidence. These results suggest prompt engineering must address both accuracy and uncertainty to be effective in high-stakes medical tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 22 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

14 of 22 references.

References · 14

  1. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  2. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  3. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  4. 2024To Believe or Not to Believe Your LLMcited by 120
  5. 2023Do Language Models Know When They're Hallucinating References?cited by 116
  6. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 105
  7. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 90
  8. 2024Uncertainty Quantification for In-Context Learning of Large Language Modelscited by 45
  9. 2024Examining the robustness of LLM evaluation to the distributional assumptions of benchmarkscited by 31
  10. 2024Cycles of Thought: Measuring LLM Confidence through Stable Explanationscited by 29
  11. 2024Just rephrase it! Uncertainty estimation in closed-source language models via multiple rephrased queriescited by 15
  12. 2025The challenge of uncertainty quantification of large language models in medicinecited by 14
2 more
  1. 2023Reflective Artificial Intelligencecited by 4
  2. 2024Accuracy and Consistency of LLMs in the Registered Dietitian Exam: The Impact of Prompt Engineering and Knowledge Retrievalcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Multi-faceted Analysis of Cognitive Abilities: Evaluating Prompt Methods with Large Language Models on the CONSORT Checklist
  2. 2026When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models
  3. 2024Enhancing Healthcare LLM Trust with Atypical Presentations Recalibration
  4. 2026When Large Language Models Fail in Healthcare: Evaluating Sensitivity to Prompt Variations
  5. 2026Confidence Calibration for Multimodal LLMs: An Empirical Study through Medical VQA
  6. 2024Bias patterns in the application of LLMs for clinical decision support: A comprehensive study
  7. 2025AutoMedPrompt: A New Framework for Optimizing LLM Medical Prompts Using Textual Gradients
  8. 2023An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing