# Few shot chain-of-thought driven reasoning to prompt LLMs for open ended medical question answering

arXiv: https://arxiv.org/abs/2403.04890 · HTML: https://mlsanity.com/paper/2403.04890/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 7 Mar 2024.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 14 of 14)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Can large language models reason about medical questions?](https://mlsanity.com/paper/2207.08143/) (2022)
- [Medical Exam Question Answering with Large-scale Reading Comprehension](https://mlsanity.com/paper/1802.10279/) (2018)
- [MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answering](https://mlsanity.com/paper/2309.16035/) (2023)
- [Explainable AI for clinical risk prediction: a survey of concepts, methods, and modalities](https://mlsanity.com/paper/2308.08407/) (2023)

## Built on by (most cited first, 30 of 31)

- [AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments](https://mlsanity.com/paper/2405.07960/) (2024)
- [Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Models](https://mlsanity.com/paper/2502.13260/) (2025)
- [On the Impact of Fine-Tuning on Chain-of-Thought Reasoning](https://mlsanity.com/paper/2411.15382/) (2024)
- [TWIN-GPT: Digital Twins for Clinical Trials via Large Language Model](https://mlsanity.com/paper/2404.01273/) (2024)
- [MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosis](https://mlsanity.com/paper/2502.19175/) (2025)
- [KGARevion: An AI Agent for Knowledge-Intensive Biomedical QA](https://mlsanity.com/paper/2410.04660/) (2024)
- [NutriBench: A Dataset for Evaluating Large Language Models on Nutrition Estimation from Meal Descriptions](https://mlsanity.com/paper/2407.12843/) (2024)
- [Why Chain of Thought Fails in Clinical Text Understanding](https://mlsanity.com/paper/2509.21933/) (2025)
- [Can LLMs Correct Physicians, Yet? Investigating Effective Interaction Methods in the Medical Domain](https://mlsanity.com/paper/2403.20288/) (2024)
- [From Data to Commonsense Reasoning: The Use of Large Language Models for Explainable AI](https://mlsanity.com/paper/2407.03778/) (2024)
- [From Intention To Implementation: Automating Biomedical Research via LLMs](https://mlsanity.com/paper/2412.09429/) (2024)
- [KnowGuard: Knowledge-Driven Abstention for Multi-Round Clinical Reasoning](https://mlsanity.com/paper/2509.24816/) (2025)
- [An Empirical Evaluation of Large Language Models on Consumer Health Questions](https://mlsanity.com/paper/2501.00208/) (2024)
- [PediaBench: A Comprehensive Chinese Pediatric Dataset for Benchmarking Large Language Models](https://mlsanity.com/paper/2412.06287/) (2024)
- [MediQAl: A French Medical Question Answering Dataset for Knowledge and Reasoning Evaluation](https://mlsanity.com/paper/2507.20917/) (2025)
- [XAI4LLM. Let Machine Learning Models and LLMs Collaborate for Enhanced In-Context Learning in Healthcare](https://mlsanity.com/paper/2405.06270/) (2024)
- [Symbolic Analysis of Grover Search Algorithm via Chain-of-Thought Reasoning and Quantum-Native Tokenization](https://mlsanity.com/paper/2505.04880/) (2025)
- [MedMKEB: A Comprehensive Knowledge Editing Benchmark for Medical Multimodal Large Language Models](https://mlsanity.com/paper/2508.05083/) (2025)
- [Towards Trustworthy Dermatology MLLMs: A Benchmark and Multimodal Evaluator for Diagnostic Narratives](https://mlsanity.com/paper/2511.09195/) (2025)
- [Generative LLM Powered Conversational AI Application for Personalized Risk Assessment: A Case Study in COVID-19](https://mlsanity.com/paper/2409.15027/) (2024)
- [Enigme: Generative Text Puzzles for Evaluating Reasoning in Language Models](https://mlsanity.com/paper/2505.04914/) (2025)
- [Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMs](https://mlsanity.com/paper/2510.20001/) (2025)
- [Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning](https://mlsanity.com/paper/2607.07761/) (2026)
- [DGRC: An Effective Fine-tuning Framework for Distractor Generation in Chinese Multi-choice Reading Comprehension](https://mlsanity.com/paper/2405.19139/) (2024)
- [CryptoLLM: Unleashing the Power of Prompted LLMs for SmartQnA and Classification of Crypto Posts](https://mlsanity.com/paper/2411.07917/) (2024)
- [HIVMedQA: Benchmarking large language models for HIV medical decision support](https://mlsanity.com/paper/2507.18143/) (2025)
- [MAC: Masked Agent Collaboration Boosts Large Language Model Medical Decision-Making](https://mlsanity.com/paper/2507.21159/) (2025)
- [Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering](https://mlsanity.com/paper/2510.18691/) (2025)
- [Leveraging Evidence-Guided LLMs to Enhance Trustworthy Depression Diagnosis](https://mlsanity.com/paper/2511.17947/) (2025)
- [Cognitive bias in LLM reasoning compromises interpretation of clinical oncology notes](https://mlsanity.com/paper/2511.20680/) (2025)

## Abstract (arXiv)

In this paper, we propose a modified version of the MedQA-USMLE dataset, named MEDQA-OPEN, which contains open-ended medical questions without options to mimic clinical scenarios, along with clinician-approved reasoned answers. Additionally, we implement a prompt driven by Chain of Thought (CoT) reasoning, CLINICR, to mirror the prospective process of incremental reasoning, reaching a correct response to medical questions. We empirically demonstrate how CLINICR outperforms the state-of-the-art 5-shot CoT-based prompt (Li'evin et al., 2022). We also present an approach that mirrors real-life clinical practice by first exploring multiple differential diagnoses through MCQ-CLINICR and subsequently narrowing down to a final diagnosis using MCQ-ELIMINATIVE. Finally, emphasizing the importance of response verification in medical settings, we utilize a reward model mechanism, replacing the elimination process performed by MCQ-ELIMINATIVE.