mlsanityData to 7 Oct 2026

Few shot chain-of-thought driven reasoning to prompt LLMs for open ended medical question answering

arXiv 2403.04890 ↗First posted 7 Mar 2024cs.CLcited by 31

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

In this paper, we propose a modified version of the MedQA-USMLE dataset, named MEDQA-OPEN, which contains open-ended medical questions without options to mimic clinical scenarios, along with clinician-approved reasoned answers. Additionally, we implement a prompt driven by Chain of Thought (CoT) reasoning, CLINICR, to mirror the prospective process of incremental reasoning, reaching a correct response to medical questions. We empirically demonstrate how CLINICR outperforms the state-of-the-art 5-shot CoT-based prompt (Li'evin et al., 2022). We also present an approach that mirrors real-life clinical practice by first exploring multiple differential diagnoses through MCQ-CLINICR and subsequently narrowing down to a final diagnosis using MCQ-ELIMINATIVE. Finally, emphasizing the importance of response verification in medical settings, we utilize a reward model mechanism, replacing the elimination process performed by MCQ-ELIMINATIVE.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 14 of its 25 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

14 of 25 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Can large language models reason about medical questions?17 Jul

Other references · 13

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  7. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  8. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  9. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  10. 2021A Survey on Automated Fact-Checkingcited by 376
  11. 2018Medical Exam Question Answering with Large-scale Reading Comprehensioncited by 29
  12. 2023MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answeringcited by 23
1 more
  1. 2023Explainable AI for clinical risk prediction: a survey of concepts, methods, and modalitiescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 15; 2026: 2.

31 papers.

Sort:

By area:

  1. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  2. 2025Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Modelscited by 51
  3. 2024On the Impact of Fine-Tuning on Chain-of-Thought Reasoningcited by 32
  4. 2024TWIN-GPT: Digital Twins for Clinical Trials via Large Language Modelcited by 24
  5. 2025MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosiscited by 23
  6. 2024KGARevion: An AI Agent for Knowledge-Intensive Biomedical QAcited by 20
  7. 2024NutriBench: A Dataset for Evaluating Large Language Models on Nutrition Estimation from Meal Descriptionscited by 17
  8. 2025Why Chain of Thought Fails in Clinical Text Understandingcited by 8
  9. 2024Can LLMs Correct Physicians, Yet? Investigating Effective Interaction Methods in the Medical Domaincited by 7
  10. 2024From Data to Commonsense Reasoning: The Use of Large Language Models for Explainable AIcited by 7
  11. 2024From Intention To Implementation: Automating Biomedical Research via LLMscited by 7
  12. 2024An Empirical Evaluation of Large Language Models on Consumer Health Questionscited by 6
19 more
  1. 2024PediaBench: A Comprehensive Chinese Pediatric Dataset for Benchmarking Large Language Modelscited by 5
  2. 2025MediQAl: A French Medical Question Answering Dataset for Knowledge and Reasoning Evaluationcited by 5
  3. 2025KnowGuard: Knowledge-Driven Abstention for Multi-Round Clinical Reasoningcited by 5
  4. 2024XAI4LLM. Let Machine Learning Models and LLMs Collaborate for Enhanced In-Context Learning in Healthcarecited by 2
  5. 2025Symbolic Analysis of Grover Search Algorithm via Chain-of-Thought Reasoning and Quantum-Native Tokenizationcited by 2
  6. 2025MedMKEB: A Comprehensive Knowledge Editing Benchmark for Medical Multimodal Large Language Modelscited by 2
  7. 2025Towards Trustworthy Dermatology MLLMs: A Benchmark and Multimodal Evaluator for Diagnostic Narrativescited by 2
  8. 2024Generative LLM Powered Conversational AI Application for Personalized Risk Assessment: A Case Study in COVID-19cited by 1
  9. 2025Enigme: Generative Text Puzzles for Evaluating Reasoning in Language Modelscited by 1
  10. 2025Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMscited by 1
  11. 2026Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoningcited by 1
  12. 2024DGRC: An Effective Fine-tuning Framework for Distractor Generation in Chinese Multi-choice Reading Comprehensioncited by 0
  13. 2024CryptoLLM: Unleashing the Power of Prompted LLMs for SmartQnA and Classification of Crypto Postscited by 0
  14. 2025HIVMedQA: Benchmarking large language models for HIV medical decision supportcited by 0
  15. 2025MAC: Masked Agent Collaboration Boosts Large Language Model Medical Decision-Makingcited by 0
  16. 2025Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answeringcited by 0
  17. 2025Leveraging Evidence-Guided LLMs to Enhance Trustworthy Depression Diagnosiscited by 0
  18. 2025Cognitive bias in LLM reasoning compromises interpretation of clinical oncology notescited by 0
  19. 2026AeroSpectra Sentinel: An Auditable LLM Prompt-Chaining Decision-Support Workflow for Acute Asthma Risk Assessment from Respiratory Sounds and Clinical Signalscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026To Reason or Not to: Selective Chain-of-Thought in Medical Question Answering
  2. 2025Instruction Tuning and CoT Prompting for Contextual Medical QA with LLMs
  3. 2024LLM-MedQA: Enhancing Medical Question Answering through Case Studies in Large Language Models
  4. 2026When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models
  5. 2024OpenMedLM: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models
  6. 2025Structured Outputs Enable General-Purpose LLMs to be Medical Experts
  7. 2025MedOmni-45deg: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine
  8. 2026Step-CoT: Stepwise Visual Chain-of-Thought for Medical Visual Question Answering