mlsanityData to 7 Oct 2026

Large Language Models Encode Clinical Knowledge

arXiv 2212.13138 ↗First posted 26 Dec 2022cs.CLcited by 1,807

Not on the map. Why?

Cited by 158 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities in natural language understanding and generation, but the quality bar for medical and clinical applications is high. Today, attempts to assess models' clinical knowledge typically rely on automated evaluations on limited benchmarks. There is no standard to evaluate model predictions and reasoning across a breadth of tasks. To address this, we present MultiMedQA, a benchmark combining six existing open question answering datasets spanning professional medical exams, research, and consumer queries; and HealthSearchQA, a new free-response dataset of medical questions searched online. We propose a framework for human evaluation of model answers along multiple axes including factuality, precision, possible harm, and bias. In addition, we evaluate PaLM (a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM, on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA, MedMCQA, PubMedQA, MMLU clinical topics), including 67.6% accuracy on MedQA (US Medical License Exam questions), surpassing prior state-of-the-art by over 17%. However, human evaluation reveals key gaps in Flan-PaLM responses. To resolve this we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, recall of knowledge, and medical reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal important limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLM models for clinical applications.

Key facts. Cited by 1,807 papers in the map (data through 7 Oct 2026). Builds on 67 of its 113 references that are in the map. First posted on arXiv 26 Dec 2022.

Builds on

67 of 113 references.

References · 67

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
38 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  13. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  14. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  15. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  16. 2018Datasheets for Datasetscited by 1,371
  17. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  18. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  19. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  20. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  21. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  22. 2018Model Cards for Model Reportingcited by 1,068
  23. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  24. 2021Ethical and social risks of harm from Language Modelscited by 957
  25. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  26. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  27. 2022Galactica: A Large Language Model for Sciencecited by 736
  28. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  29. 2021GPT Understands, Toocited by 715
  30. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  31. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  32. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  33. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 380
  34. 2022Can language models learn from explanations in context?cited by 248
  35. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  36. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  37. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  38. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 285; 2024: 517; 2025: 504; 2026: 500.

1,807 papers.

Sort:

By area:

  1. 2024GPT-4o System Cardcited by 4,467
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  4. 2023BloombergGPT: A Large Language Model for Financecited by 923
  5. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  6. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  7. 2024Large Language Models: A Surveycited by 506
  8. 2025Humanity's Last Examcited by 467
  9. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  10. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  11. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  12. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
38 more
  1. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 333
  2. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  3. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 319
  4. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  5. 2023Towards Generalist Biomedical AIcited by 286
  6. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  7. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  8. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  9. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  10. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  11. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  12. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  13. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  14. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  15. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  16. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  17. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  18. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  19. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  20. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  21. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  22. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  23. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
  24. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
  25. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  26. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
  27. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
  28. 2023Med-HALT: Medical Domain Hallucination Test for Large Language Modelscited by 156
  29. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  30. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 151
  31. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 145
  32. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144
  33. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
  34. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 142
  35. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  36. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 138
  37. 2023On the Challenges and Perspectives of Foundation Models for Medical Image Analysiscited by 135
  38. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134

50 most cited of 1,807 All 1,807 (JSON)