mlsanityData to 14 Oct 2026

A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment

arXiv 2505.10717 ↗First posted 15 May 2025cs.CL cs.AIcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Medical LLM alignment

also in 1 more
Abstract (arXiv)

High computation costs and latency of large language models such as GPT-4 have limited their deployment in clinical settings. Small language models (SLMs) offer a cost-effective alternative, but their limited capacity requires biomedical domain adaptation, which remains challenging. An additional bottleneck is the unavailability and high sensitivity of clinical data. To address these challenges, we propose a novel framework for adapting SLMs into high-performing clinical models. We introduce the MediPhi collection of 3.8B-parameter SLMs developed with our novel framework: pre-instruction tuning of experts on relevant medical and clinical corpora (PMC, Medical Guideline, MedWiki, etc.), model merging, and clinical-tasks alignment. To cover most clinical tasks, we extended the CLUE benchmark to CLUE+, doubling its size. Our expert models deliver relative improvements on this benchmark over the base model without any task-specific fine-tuning: 64.3% on medical entities, 49.5% on radiology reports, and 44% on ICD-10 coding (outperforming GPT-4-0125 by 14%). We unify the expert models into MediPhi via model merging, preserving gains across benchmarks. Furthermore, we built the MediFlow collection, a synthetic dataset of 2.5 million high-quality instructions on 14 medical NLP tasks, 98 fine-grained document types, and JSON format support. Alignment of MediPhi using supervised fine-tuning and direct preference optimization achieves further gains of 18.9% on average.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Medical LLM alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 52 of its 87 references that are in the map. First posted on arXiv 15 May 2025.

Builds on

52 of 87 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023AlpaCare:Instruction-tuned Large Language Models for Medical Application23 Oct

Other references · 51

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  9. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  10. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  11. 2022Editing Models with Task Arithmeticcited by 1,120
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
38 more
  1. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  2. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  3. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  4. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  5. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  6. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  7. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  8. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  9. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  10. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
  11. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  12. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  13. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
  14. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  15. 2022Fine-tuned Language Models are Continual Learnerscited by 156
  16. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 151
  17. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 145
  18. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 133
  19. 2023ACI-BENCH: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generationcited by 118
  20. 2024Med42-v2: A Suite of Clinical LLMscited by 90
  21. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 89
  22. 2024MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notescited by 80
  23. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  24. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 55
  25. 2024What Matters for Model Merging at Scale?cited by 55
  26. 2023Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notescited by 51
  27. 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 46
  28. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 45
  29. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunchcited by 43
  30. 2024LongHealth: A Question Answering Benchmark with Long Clinical Documentscited by 33
  31. 2024ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?cited by 32
  32. 2024From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyondcited by 24
  33. 2024Aloe: A Family of Fine-tuned Open Healthcare LLMscited by 20
  34. 2024Hippocrates: An Open-Source Framework for Advancing Large Language Models in Healthcarecited by 15
  35. 2024Does Biomedical Training Lead to Better Medical Performance?cited by 13
  36. 2024Scalable Data Ablation Approximations for Language Models through Modular Training and Mergingcited by 11
  37. 2024Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMscited by 8
  38. 2024IryoNLP at MEDIQA-CORR 2024: Tackling the Medical Error Detection & Correction Task On the Shoulders of Medical Agentscited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 11.

20 papers.

Sort:

By area:

  1. 2025Understanding the Effects of Domain Finetuning on LLMscited by 4
  2. 2025Enhancing the Medical Context-Awareness Ability of LLMs via Multifaceted Self-Refinement Learningcited by 4
  3. 2025Understanding Post-Training Structural Changes in Large Language Modelscited by 3
  4. 2025MedRiskEval: Medical Risk Evaluation Benchmark of Language Models, On the Importance of User Perspectives in Healthcare Settingscited by 2
  5. 2025Large language models surpass domain-specific architectures for antepartum electronic fetal monitoring analysiscited by 1
  6. 2025H-DDx: A Hierarchical Evaluation Framework for Differential Diagnosiscited by 1
  7. 2026A Causal Language Modeling Detour Improves Encoder Continued Pretrainingcited by 1
  8. 2026Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?cited by 1
  9. 2025Retrieval Augmented Large Language Model System for Comprehensive Drug Contraindicationscited by 0
  10. 2025Additive Large Language Models for Semi-Structured Textcited by 0
  11. 2025Leveraging LLMs for Title and Abstract Screening for Systematic Review: A Cost-Effective Dynamic Few-Shot Learning Approachcited by 0
  12. 2026Modular Expert Merging for Biomedical Retrievalcited by 0
8 more
  1. 2026How Robust Are Large Language Models for Clinical Numeracy? An Empirical Study on Numerical Reasoning Abilities in Clinical Contextscited by 0
  2. 2026Reliable Multilingual Orthopedic Decision Support from Clinical Narratives: Language-Aware Adaptation and Verification-Guided Deferralcited by 0
  3. 2026DECSELFMASK: Leveraging Unlabeled Text via Self-Relevance-Guided Masking for Decoder-Only Classificationcited by 0
  4. 2026eCREAM-MedCorpus A Large-Scale Corpus of Clinical Notes for Italiancited by 0
  5. 2026Position: Medical AI Neglects Real Treatment Outcomescited by 0
  6. 2026Toward Better Assessment of LLMs' Performance in Clinical Error Detectioncited by 0
  7. 2026HealMed: Multilingual Evaluation of Large Language Models in Medicinecited by 0
  8. 2026Aligning with Lived Experience: Heterogeneous Benefits of Fine Tuning in Mental Health Support Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data
  2. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
  3. 2024MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation
  4. 2025Evaluating Vision Language Model Adaptations for Radiology Report Generation in Low-Resource Languages
  5. 2025Benchmarking and Adapting On-Device LLMs for Clinical Decision Support
  6. 2026To Adapt or not to Adapt, Rethinking the Value of Medical Knowledge-Aware Large Language Models
  7. 2023Exploring the Trade-Offs: Unified Large Language Models vs Local Fine-Tuned Models for Highly-Specific Radiology NLI Task
  8. 2024BioMamba: Domain-Adaptive Biomedical Language Models