mlsanityData to 7 Oct 2026

Towards Expert-Level Medical Question Answering with Large Language Models

arXiv 2305.09617 ↗First posted 16 May 2023cs.CL cs.AI cs.LGcited by 382

Not on the map. Why?

Cited by 42 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent artificial intelligence (AI) systems have reached milestones in "grand challenges" ranging from Go to protein-folding. The capability to retrieve medical knowledge, reason over it, and answer medical questions comparably to physicians has long been viewed as one such grand challenge. Large language models (LLMs) have catalyzed significant progress in medical question answering; Med-PaLM was the first model to exceed a "passing" score in US Medical Licensing Examination (USMLE) style questions with a score of 67.2% on the MedQA dataset. However, this and other prior work suggested significant room for improvement, especially when models' answers were compared to clinicians' answers. Here we present Med-PaLM 2, which bridges these gaps by leveraging a combination of base LLM improvements (PaLM 2), medical domain finetuning, and prompting strategies including a novel ensemble refinement approach. Med-PaLM 2 scored up to 86.5% on the MedQA dataset, improving upon Med-PaLM by over 19% and setting a new state-of-the-art. We also observed performance approaching or exceeding state-of-the-art across MedMCQA, PubMedQA, and MMLU clinical topics datasets. We performed detailed human evaluations on long-form questions along multiple axes relevant to clinical applications. In pairwise comparative ranking of 1066 consumer medical questions, physicians preferred Med-PaLM 2 answers to those produced by physicians on eight of nine axes pertaining to clinical utility (p < 0.001). We also observed significant improvements compared to Med-PaLM on every evaluation axis (p < 0.001) on newly introduced datasets of 240 long-form "adversarial" questions to probe LLM limitations. While further studies are necessary to validate the efficacy of these models in real-world settings, these results highlight rapid progress towards physician-level performance in medical question answering.

Key facts. Cited by 382 papers in the map (data through 7 Oct 2026). Builds on 27 of its 51 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

27 of 51 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  11. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  12. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
15 more
  1. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  2. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  3. 2022Red Teaming Language Models with Language Modelscited by 983
  4. 2021Ethical and social risks of harm from Language Modelscited by 957
  5. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  6. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  7. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  8. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  9. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  10. 2022Can large language models reason about medical questions?cited by 194
  11. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  12. 2020Ethical Machine Learning in Health Carecited by 112
  13. 2022Recitation-Augmented Language Modelscited by 65
  14. 2023DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agentscited by 52
  15. 2022Active Acquisition for Multimodal Temporal Data: A Challenging Decision-Making Taskcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 92; 2024: 176; 2025: 79; 2026: 35.

382 papers.

Sort:

By area:

  1. 2024GPT-4o System Cardcited by 4,467
  2. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  3. 2024Large Language Models: A Surveycited by 506
  4. 2023Llemma: An Open Language Model For Mathematicscited by 385
  5. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  6. 2023Towards Generalist Biomedical AIcited by 286
  7. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  8. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  9. 2023Challenges and Applications of Large Language Modelscited by 251
  10. 2024Capabilities of Gemini Models in Medicinecited by 231
  11. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 211
  12. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
38 more
  1. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  2. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  3. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
  4. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  5. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  6. 2023Med-HALT: Medical Domain Hallucination Test for Large Language Modelscited by 156
  7. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  8. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  9. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  10. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 145
  11. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  12. 2023On the Challenges and Perspectives of Foundation Models for Medical Image Analysiscited by 135
  13. 2024LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Modelcited by 133
  14. 2023Large language models in healthcare and medical domain: A reviewcited by 126
  15. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  16. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  17. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  18. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  19. 2023Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialoguecited by 118
  20. 2024Empowering Biomedical Discovery with AI Agentscited by 117
  21. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111
  22. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  23. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMscited by 103
  24. 2024Towards Building Multilingual Language Model for Medicinecited by 91
  25. 2023DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultationcited by 89
  26. 2023Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunitiescited by 89
  27. 2024Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generationcited by 89
  28. 2024Med42-v2: A Suite of Clinical LLMscited by 89
  29. 2024Advancing Multimodal Medical Capabilities of Geminicited by 88
  30. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  31. 2023Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicinecited by 83
  32. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  33. 2024Towards Conversational Diagnostic AIcited by 79
  34. 2024MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notescited by 77
  35. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 74
  36. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 74
  37. 2024Safeguarding Large Language Models: A Surveycited by 73
  38. 2023Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encodingcited by 71

50 most cited of 382 All 382 (JSON)