mlsanityData to 13 Oct 2026

Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs

arXiv 2409.14988 ↗First posted 23 Sep 2024cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated significant potential in transforming clinical applications. In this study, we investigate the efficacy of four techniques in adapting LLMs for clinical use-cases: continuous pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning dataset of 500 million tokens. Our evaluation across various clinical tasks reveals the impact of each technique. While continuous pretraining beyond 250 billion tokens yields marginal improvements on its own, it establishes a strong foundation for instruct fine-tuning. Notably, NEFTune, designed primarily to enhance generation quality, surprisingly demonstrates additional gains on our benchmark. Complex prompt engineering methods further enhance performance. These findings show the importance of tailoring fine-tuning strategies and exploring innovative techniques to optimize LLM performance in the clinical domain.

Key facts. Cited by 8 papers in the map (data through 13 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

29 of 44 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,348
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  5. 2022Large Language Models Encode Clinical Knowledgecited by 1,817
  6. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,338
  7. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,281
  8. 2023LIMA: Less Is More for Alignmentcited by 1,091
  9. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 865
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 833
  11. 2017Crowdsourcing Multiple Choice Science Questionscited by 751
  12. 2024Yi: Open Foundation Models by 01.AIcited by 715
17 more
  1. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 714
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 710
  3. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 635
  4. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 564
  5. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  6. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 324
  7. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 285
  8. 2024Capabilities of Gemini Models in Medicinecited by 236
  9. 2020Explainable Automated Fact-Checking for Public Health Claimscited by 211
  10. 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
  11. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 137
  12. 2019A Question-Entailment Approach to Question Answeringcited by 134
  13. 2019HEAD-QA: A Healthcare Dataset for Complex Reasoningcited by 101
  14. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 92
  15. 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 87
  16. 2023Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encodingcited by 71
  17. 2024Med42 -- Evaluating Fine-Tuning Strategies for Medical LLMs: Full-Parameter vs. Parameter-Efficient Approachescited by 52

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  2. 2025Format Inertia: A Failure Mechanism of LLMs in Medical Pre-Consultationcited by 1
  3. 2026MedInjection-FR: Exploring the Role of Native, Synthetic, and Translated Data in Biomedical Instruction Tuningcited by 1
  4. 2026DKL: Decoupled Knowledge Learning for Instruction-Tuned Language Modelscited by 1
  5. 2026Trade-offs in Medical LLM Adaptation: An Empirical Study in French QAcited by 0
  6. 2026KItCAT: Knowledge Injection via Input Corruption for Auto-regressive Trainingcited by 0
  7. 2026Didactic knowledge or Clinical Cases? How Data Types Shape Medical Large Language Modelscited by 0
  8. 2026Localization Lens for Improving Medical Vision-Language Modelscited by 0