# Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine

arXiv: https://arxiv.org/abs/2308.06834 · HTML: https://mlsanity.com/paper/2308.06834/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 11 of its 34 references that are in the map.
- First posted on arXiv 13 Aug 2023.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [Prompt Engineering for Healthcare: Methodologies and Applications](https://mlsanity.com/paper/2304.14670/) (2023)

## Built on by (most cited first, 30 of 83)

- [Large language models in healthcare and medical domain: A review](https://mlsanity.com/paper/2401.06775/) (2023)
- [Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation](https://mlsanity.com/paper/2408.04187/) (2024)
- [MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes](https://mlsanity.com/paper/2412.19260/) (2024)
- [Evaluating Step-by-step Reasoning Traces: A Survey](https://mlsanity.com/paper/2502.12289/) (2025)
- [Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questions](https://mlsanity.com/paper/2408.00727/) (2024)
- [CoD, Towards an Interpretable Medical Agent using Chain of Diagnosis](https://mlsanity.com/paper/2407.13301/) (2024)
- [Medical Large Language Model Benchmarks Should Prioritize Construct Validity](https://mlsanity.com/paper/2503.10694/) (2025)
- [MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports](https://mlsanity.com/paper/2505.11733/) (2025)
- [Superhuman performance of a large language model on the reasoning tasks of a physician](https://mlsanity.com/paper/2412.10849/) (2024)
- [MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosis](https://mlsanity.com/paper/2502.19175/) (2025)
- [Hyper-RAG: Combating LLM Hallucinations using Hypergraph-Driven Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.08758/) (2025)
- [Automating Expert-Level Medical Reasoning Evaluation of Large Language Models](https://mlsanity.com/paper/2507.07988/) (2025)
- [Prompt engineering paradigms for medical applications: scoping review and recommendations for better practices](https://mlsanity.com/paper/2405.01249/) (2024)
- [ArgMed-Agents: Explainable Clinical Decision Reasoning with LLM Disscusion via Argumentation Schemes](https://mlsanity.com/paper/2403.06294/) (2024)
- [BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text](https://mlsanity.com/paper/2504.19467/) (2025)
- [Me LLaMA: Foundation Large Language Models for Medical Applications](https://mlsanity.com/paper/2402.12749/) (2024)
- [Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis](https://mlsanity.com/paper/2403.16303/) (2024)
- [A Survey of Scaling in Large Language Model Reasoning](https://mlsanity.com/paper/2504.02181/) (2025)
- [Uncertainty-Aware Large Language Models for Explainable Disease Diagnosis](https://mlsanity.com/paper/2505.03467/) (2025)
- [Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations](https://mlsanity.com/paper/2510.11196/) (2025)
- [Generalizable and Scalable Multistage Biomedical Concept Normalization Leveraging Large Language Models](https://mlsanity.com/paper/2405.15122/) (2024)
- [When Reasoning Meets Information Aggregation: A Case Study with Sports Narratives](https://mlsanity.com/paper/2406.12084/) (2024)
- [Integrating Chain-of-Thought and Retrieval Augmented Generation Enhances Rare Disease Diagnosis from Clinical Notes](https://mlsanity.com/paper/2503.12286/) (2025)
- [ClinicalGPT-R1: Pushing reasoning capability of generalist disease diagnosis with large language model](https://mlsanity.com/paper/2504.09421/) (2025)
- [MedKGI: Iterative Differential Diagnosis with Medical Knowledge Graphs and Information-Guided Inquiring](https://mlsanity.com/paper/2512.24181/) (2025)
- [Fine Tuning Large Language Models for Medicine: The Role and Importance of Direct Preference Optimization](https://mlsanity.com/paper/2409.12741/) (2024)
- [MACD: Multi-Agent Clinical Diagnosis with Self-Learned Knowledge for LLM](https://mlsanity.com/paper/2509.20067/) (2025)
- [Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questions](https://mlsanity.com/paper/2404.13307/) (2024)
- [Best Practices for Large Language Models in Radiology](https://mlsanity.com/paper/2412.01233/) (2024)
- [Large Language Models for Medical Forecasting -- Foresight 2](https://mlsanity.com/paper/2412.10848/) (2024)

## Abstract (arXiv)

One of the major barriers to using large language models (LLMs) in medicine is the perception they use uninterpretable methods to make clinical decisions that are inherently different from the cognitive processes of clinicians. In this manuscript we develop novel diagnostic reasoning prompts to study whether LLMs can perform clinical reasoning to accurately form a diagnosis. We find that GPT4 can be prompted to mimic the common clinical reasoning processes of clinicians without sacrificing diagnostic accuracy. This is significant because an LLM that can use clinical reasoning to provide an interpretable rationale offers physicians a means to evaluate whether LLMs can be trusted for patient care. Novel prompting methods have the potential to expose the black box of LLMs, bringing them one step closer to safe and effective use in medicine.