mlsanityData to 7 Oct 2026

Can large language models reason about medical questions?

arXiv 2207.08143 ↗First posted 17 Jul 2022cs.CL cs.AI cs.LGcited by 194

On the map: LLM reasoning → Clinical and biomedical applications

Abstract (arXiv)

Although large language models (LLMs) often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge. We set out to investigate whether close- and open-source models (GPT-3.5, LLama-2, etc.) can be applied to answer and reason about difficult real-world-based questions. We focus on three popular medical benchmarks (MedQA-USMLE, MedMCQA, and PubMedQA) and multiple prompting scenarios: Chain-of-Thought (CoT, think step-by-step), few-shot and retrieval augmentation. Based on an expert annotation of the generated CoTs, we found that InstructGPT can often read, reason and recall expert knowledge. Last, by leveraging advances in prompt engineering (few-shot and ensemble methods), we demonstrated that GPT-3.5 not only yields calibrated predictive distributions, but also reaches the passing score on three datasets: MedQA-USMLE 60.2%, MedMCQA 62.7% and PubMedQA 78.2%. Open-source models are closing the gap: Llama-2 70B also passed the MedQA-USMLE with 62.5% accuracy.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning. Builds on 38 of its 80 references that are in the map. First posted on arXiv 17 Jul 2022.

Builds on

38 of 80 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
26 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  12. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  13. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  14. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  15. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  16. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  17. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  18. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  19. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  20. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  21. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  22. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  23. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  24. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  25. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  26. 2021GPT-3 Models are Poor Few-Shot Learners in the Biomedical Domaincited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 64; 2024: 58; 2025: 46; 2026: 22.

194 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  4. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  5. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  6. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  7. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  8. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  9. 2023Challenges and Applications of Large Language Modelscited by 251
  10. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  11. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  12. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 210
38 more
  1. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  2. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  3. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  4. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  5. 2023A Categorical Archive of ChatGPT Failurescited by 167
  6. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  7. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  8. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  9. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicinecited by 122
  10. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  11. 2023GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Informationcited by 115
  12. 2024Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questionscited by 106
  13. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMscited by 103
  14. 2023Do We Still Need Clinical Language Models?cited by 97
  15. 2024CRAG -- Comprehensive RAG Benchmarkcited by 94
  16. 2023Unifying Molecular and Textual Representations via Multi-task Language Modellingcited by 92
  17. 2022ChatGPT: The End of Online Exam Integrity?cited by 88
  18. 2023Can large language models build causal graphs?cited by 72
  19. 2023Matching Patients to Clinical Trials with Large Language Modelscited by 68
  20. 2024MedCalc-Bench: Evaluating Large Language Models for Medical Calculationscited by 64
  21. 2023Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Taskscited by 62
  22. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  23. 2023Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?cited by 54
  24. 2024Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questionscited by 54
  25. 2023DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agentscited by 52
  26. 2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
  27. 2023Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notescited by 51
  28. 2024A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Taskscited by 46
  29. 2023Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success)cited by 40
  30. 2025Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactionscited by 40
  31. 2023Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Coursescited by 38
  32. 2024Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B Peoplecited by 38
  33. 2025MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworkscited by 38
  34. 2024Addressing cognitive bias in medical language modelscited by 34
  35. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
  36. 2024EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summariescited by 33
  37. 2025TimeCAP: Learning to Contextualize, Augment, and Predict Time Series Events with Large Language Model Agentscited by 33
  38. 2023Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Codecited by 31

50 most cited of 194 All 194 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Disentangling Reasoning and Knowledge in Medical Large Language Models
  2. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering
  3. 2024Pattern Recognition or Medical Knowledge? The Problem with Multiple-Choice Questions in Medicine
  4. 2025Evaluating Large Language Models for Evidence-Based Clinical Question Answering
  5. 2023Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset
  6. 2026Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation
  7. 2025Multilingual Medical Reasoning for Question Answering with Large Language Models
  8. 2023ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation