mlsanityData to 7 Oct 2026

MEDITRON-70B: Scaling Medical Pretraining for Large Language Models

arXiv 2311.16079 ↗First posted 27 Nov 2023cs.CL cs.AI cs.LGcited by 272

Not on the map. Why?

Cited by 59 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) can potentially democratize access to medical knowledge. While many efforts have been made to harness and improve LLMs' medical knowledge and reasoning capacities, the resulting models are either closed-source (e.g., PaLM, GPT-4) or limited in scale (<= 13B parameters), which restricts their abilities. In this work, we improve access to large-scale medical LLMs by releasing MEDITRON: a suite of open-source LLMs with 7B and 70B parameters adapted to the medical domain. MEDITRON builds on Llama-2 (through our adaptation of Nvidia's Megatron-LM distributed trainer), and extends pretraining on a comprehensively curated medical corpus, including selected PubMed articles, abstracts, and internationally-recognized medical guidelines. Evaluations using four major medical benchmarks show significant performance gains over several state-of-the-art baselines before and after task-specific finetuning. Overall, MEDITRON achieves a 6% absolute performance gain over the best public baseline in its parameter class and 3% over the strongest baseline we finetuned from Llama-2. Compared to closed-source LLMs, MEDITRON-70B outperforms GPT-3.5 and Med-PaLM and is within 5% of GPT-4 and 10% of Med-PaLM-2. We release our code for curating the medical pretraining corpus and the MEDITRON model weights to drive open-source development of more capable medical LLMs.

Key facts. Cited by 272 papers in the map (data through 7 Oct 2026). Builds on 48 of its 71 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

48 of 71 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
36 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  9. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  12. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  13. 2019Publicly Available Clinical BERT Embeddingscited by 995
  14. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  15. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  16. 2023StarCoder: may the source be with you!cited by 790
  17. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  18. 2022Galactica: A Large Language Model for Sciencecited by 736
  19. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  20. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  21. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  22. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  23. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  24. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  25. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  26. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
  27. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  28. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  29. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  30. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  31. 2023Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encodingcited by 71
  32. 2019IMHO Fine-Tuning Improves Claim Detectioncited by 27
  33. 2019Efficient Adaptation of Pretrained Transformers for Abstractive Summarizationcited by 13
  34. 2019Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networkscited by 12
  35. 2022Continued Pretraining for Better Zero- and Few-Shot Promptabilitycited by 10
  36. 2023FedJudge: Federated Legal Large Language Modelcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 107; 2025: 101; 2026: 63.

272 papers.

Sort:

By area:

  1. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  2. 2024AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environmentscited by 215
  3. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  4. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  5. 2024Towards Building Multilingual Language Model for Medicinecited by 91
  6. 2024Med42-v2: A Suite of Clinical LLMscited by 89
  7. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 82
  8. 2024Towards Conversational Diagnostic AIcited by 79
  9. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  10. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  11. 2024MedCalc-Bench: Evaluating Large Language Models for Medical Calculationscited by 64
  12. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 57
38 more
  1. 2025TxAgent: An AI Agent for Therapeutic Reasoning Across a Universe of Toolscited by 57
  2. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  3. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  4. 2024MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Modelscited by 43
  5. 2025Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactionscited by 40
  6. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  7. 2024Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B Peoplecited by 38
  8. 2024Towards Evaluating and Building Versatile Large Language Models for Medicinecited by 38
  9. 2025MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworkscited by 38
  10. 2024MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoningcited by 35
  11. 2024Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Modelscited by 34
  12. 2024Addressing cognitive bias in medical language modelscited by 34
  13. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  14. 2024EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summariescited by 33
  15. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?cited by 33
  16. 2025TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Recordscited by 33
  17. 2025Medical World Model: Generative Simulation of Tumor Evolution for Treatment Planningcited by 31
  18. 2024Small Language Models Learn Enhanced Reasoning Skills from Medical Textbookscited by 30
  19. 2025Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoningcited by 29
  20. 2025Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Reportcited by 28
  21. 2024Bias patterns in the application of LLMs for clinical decision support: A comprehensive studycited by 27
  22. 2024Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementationcited by 27
  23. 2025CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMscited by 26
  24. 2024Efficient Continual Pre-training by Mitigating the Stability Gapcited by 25
  25. 2024MEDIC: Comprehensive Evaluation of Leading Indicators for LLM Safety and Utility in Clinical Applicationscited by 25
  26. 2024A Benchmark for Long-Form Medical Question Answeringcited by 25
  27. 2025ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoningcited by 24
  28. 2025MA-RAG: Multi-Agent Retrieval-Augmented Generation via Collaborative Chain-of-Thought Reasoningcited by 24
  29. 2025Med-PRM: Medical Reasoning Models with Stepwise, Guideline-verified Process Rewardscited by 24
  30. 2024KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinationscited by 22
  31. 2024Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Datacited by 21
  32. 2025Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Reportcited by 21
  33. 2024OpenMedLM: Prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language modelscited by 20
  34. 2024Towards Modeling Learner Performance with Large Language Modelscited by 20
  35. 2024Aloe: A Family of Fine-tuned Open Healthcare LLMscited by 20
  36. 2024OLAPH: Improving Factuality in Biomedical Long-form Question Answeringcited by 20
  37. 2024ReflecTool: Towards Reflection-Aware Tool-Augmented Clinical Agentscited by 20
  38. 2024The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Bettercited by 19

50 most cited of 272 All 272 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MedLLM: An Open Medical Language Model at the Sub-Billion Scale
  2. 2026Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks
  3. 2024MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation
  4. 2025MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks
  5. 2025IIMedGPT: Promoting Large Language Model Capabilities of Medical Tasks by Efficient Human Preference Alignment
  6. 2023From Beginner to Expert: Modeling Medical Knowledge into General LLMs
  7. 2023ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences
  8. 2025Disentangling Reasoning and Knowledge in Medical Large Language Models