mlsanityData to 14 Oct 2026

PMC-LLaMA: Towards Building Open-source Language Models for Medicine

arXiv 2304.14454 ↗First posted 27 Apr 2023cs.CLcited by 88

Not on the map. Why?

Abstract (arXiv)

Recently, Large Language Models (LLMs) have showcased remarkable capabilities in natural language understanding. While demonstrating proficiency in everyday conversations and question-answering situations, these models frequently struggle in domains that require precision, such as medical applications, due to their lack of domain-specific knowledge. In this paper, we describe the procedure for building a powerful, open-source language model specifically designed for medicine applications, termed as PMC-LLaMA. Our contributions are threefold: (i) we systematically investigate the process of adapting a general-purpose foundation language model towards medical domain, this involves data-centric knowledge injection through the integration of 4.8M biomedical academic papers and 30K medical textbooks, as well as comprehensive fine-tuning for alignment with domain-specific instructions; (ii) we contribute a large-scale, comprehensive dataset for instruction tuning. This dataset encompasses medical question-answering (QA), rationale for reasoning, and conversational dialogues, comprising a total of 202M tokens; (iii) we conduct thorough ablation studies to demonstrate the effectiveness of each proposed component. While evaluating on various public medical question-answering benchmarks, our lightweight PMCLLaMA, which consists of only 13 billion parameters, exhibits superior performance, even surpassing ChatGPT. All models, codes, datasets can be found in https://github.com/chaoyi-wu/PMC-LLaMA.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Builds on 15 of its 29 references that are in the map. First posted on arXiv 27 Apr 2023.

Builds on

15 of 29 references.

References · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  7. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  8. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  10. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  11. 2023Instruction Tuning with GPT-4cited by 686
  12. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
3 more
  1. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  2. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 286
  3. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 48; 2025: 16; 2026: 12.

88 papers.

Sort:

By area:

  1. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  2. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  3. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  4. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 231
  5. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 154
  6. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  7. 2024Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generationcited by 90
  8. 2023InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discoverycited by 86
  9. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 77
  10. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  11. 2023LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Modelscited by 60
  12. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 56
38 more
  1. 2024Med42 -- Evaluating Fine-Tuning Strategies for Medical LLMs: Full-Parameter vs. Parameter-Efficient Approachescited by 52
  2. 2024MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Modelscited by 44
  3. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answeringcited by 43
  4. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  5. 2023Taiyi: A Bilingual Fine-Tuned Large Language Model for Diverse Biomedical Taskscited by 36
  6. 2024Addressing cognitive bias in medical language modelscited by 36
  7. 2023ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMscited by 35
  8. 2023HealthFC: Verifying Health Claims with Evidence-Based Medical Fact-Checkingcited by 35
  9. 2023OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialoguecited by 34
  10. 2024GSCo: Towards Generalizable AI in Medicine via Generalist-Specialist Collaborationcited by 34
  11. 2024JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capabilitycited by 32
  12. 2023MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Modelscited by 31
  13. 2024A Continued Pretrained LLM Approach for Automatic Medical Note Generationcited by 30
  14. 2024Small Language Models Learn Enhanced Reasoning Skills from Medical Textbookscited by 30
  15. 2024Bias patterns in the application of LLMs for clinical decision support: A comprehensive studycited by 27
  16. 2024Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementationcited by 27
  17. 2024Efficient Continual Pre-training by Mitigating the Stability Gapcited by 25
  18. 2024Exploring the Impact of Table-to-Text Methods on Augmenting LLM-based Question Answering with Domain Hybrid Datacited by 24
  19. 2023Large language models in medicine: the potentials and pitfallscited by 21
  20. 2024OpenMedLM: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language modelscited by 20
  21. 2025The Rise of Small Language Models in Healthcare: A Comprehensive Surveycited by 18
  22. 2024Prompt-based Code Completion via Multi-Retrieval Augmented Generationcited by 16
  23. 2024Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domaincited by 15
  24. 2025Knowledge-Instruct: Effective Continual Pre-training from Limited Data using Instructionscited by 12
  25. 2023From Classification to Clinical Insights: Towards Analyzing and Reasoning About Mobile and Behavioral Health Data With Large Language Modelscited by 11
  26. 2024CasiMedicos-Arg: A Medical Question Answering Dataset Annotated with Explanatory Argumentative Structurescited by 11
  27. 2024Automatically Extracting Numerical Results from Randomized Controlled Trials with Large Language Modelscited by 9
  28. 2025LExT: Towards Evaluating Trustworthiness of Natural Language Explanationscited by 9
  29. 2024BioMamba: Domain-Adaptive Biomedical Language Modelscited by 8
  30. 2024Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMscited by 8
  31. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practicecited by 8
  32. 2025Memorize and Rank: Elevating Large Language Models for Clinical Diagnosis Predictioncited by 8
  33. 2024Analysing zero-shot temporal relation extraction on clinical notes using temporal consistencycited by 7
  34. 2024MedINST: Meta Dataset of Biomedical Instructionscited by 7
  35. 2023Establishing Performance Baselines in Fine-Tuning, Retrieval-Augmented Generation and Soft-Prompting for Non-Specialist LLM Userscited by 6
  36. 2024Clinical Insights: A Comprehensive Review of Language Models in Medicinecited by 6
  37. 2025Exploring Efficiency Frontiers of Thinking Budget in Medical Reasoning: Scaling Laws between Computational Resources and Reasoning Qualitycited by 6
  38. 2023Revisiting the Knowledge Injection Frameworkscited by 5

50 most cited of 88 All 88 (JSON)