mlsanityData to 14 Oct 2026

Capabilities of GPT-4 on Medical Challenge Problems

arXiv 2303.13375 ↗First posted 20 Mar 2023cs.CL cs.AIcited by 638

Not on the map. Why?

Cited by 79 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation across various domains, including medicine. We present a comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets. GPT-4 is a general-purpose model that is not specialized for medical problems through training or engineered to solve clinical tasks. Our analysis covers two sets of official practice materials for the USMLE, a three-step examination program used to assess clinical competency and grant licensure in the United States. We also evaluate performance on the MultiMedQA suite of benchmark datasets. Beyond measuring model performance, experiments were conducted to investigate the influence of test questions containing both text and images on model performance, probe for memorization of content during training, and study probability calibration, which is of critical importance in high-stakes applications like medicine. Our results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B). In addition, GPT-4 is significantly better calibrated than GPT-3.5, demonstrating a much-improved ability to predict the likelihood that its answers are correct. We also explore the behavior of the model qualitatively through a case study that shows the ability of GPT-4 to explain medical reasoning, personalize explanations to students, and interactively craft new counterfactual scenarios around a medical case. Implications of the findings are discussed for potential uses of GPT-4 in medical education, assessment, and clinical practice, with appropriate attention to challenges of accuracy and safety.

Key facts. Cited by 638 papers in the map (data through 14 Oct 2026). Builds on 27 of its 77 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

27 of 77 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2017Deep reinforcement learning from human preferencescited by 3,912
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
15 more
  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  4. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  5. 2018A Reductions Approach to Fair Classificationcited by 587
  6. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  7. 2017CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learningcited by 461
  8. 2019Measurement and Fairnesscited by 308
  9. 2019InterpretML: A Unified Framework for Machine Learning Interpretabilitycited by 246
  10. 2022Can large language models reason about medical questions?cited by 194
  11. 2021A Universal Law of Robustness via Isoperimetrycited by 151
  12. 2020Learning to Complement Humanscited by 124
  13. 2022Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programmingcited by 91
  14. 2020An Empirical Analysis of Backward Compatibility in Machine Learning Systemscited by 38
  15. 2022Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imagingcited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 166; 2024: 199; 2025: 139; 2026: 134.

638 papers.

Sort:

By area:

  1. 2024GPT-4o System Cardcited by 4,588
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  3. 2023A Survey of Large Language Modelscited by 2,576
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  6. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  7. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  8. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  9. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 384
  10. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 353
  11. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  12. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
38 more
  1. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  2. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 316
  3. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
  4. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 274
  5. 2023How is ChatGPT's behavior changing over time?cited by 270
  6. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 270
  7. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 246
  8. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  9. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  10. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  11. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 187
  12. 2023What can Large Language Models do in chemistry? A comprehensive benchmark on eight taskscited by 185
  13. 2023SciMON: Scientific Inspiration Machines Optimized for Noveltycited by 179
  14. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 178
  15. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  16. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
  17. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 155
  18. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 124
  19. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  20. 2023GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Informationcited by 117
  21. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 116
  22. 2023The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4cited by 112
  23. 2024Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questionscited by 109
  24. 2023CLadder: Assessing Causal Reasoning in Language Modelscited by 107
  25. 2024OLMES: A Standard for Language Model Evaluationscited by 106
  26. 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 104
  27. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 98
  28. 2023LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunitiescited by 97
  29. 2023Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Datasetcited by 91
  30. 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 88
  31. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 86
  32. 2023Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicinecited by 83
  33. 2023Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queriescited by 82
  34. 2024RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculturecited by 81
  35. 2023Benchmarking large language models for biomedical natural language processing applications and recommendationscited by 78
  36. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  37. 2023Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encodingcited by 71
  38. 2024A Complete Survey on LLM-based AI Chatbotscited by 71

50 most cited of 638 All 638 (JSON)