mlsanityData to 14 Oct 2026

The Rise of Small Language Models in Healthcare: A Comprehensive Survey

arXiv 2504.17119 ↗First posted 23 Apr 2025cs.CL cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Despite substantial progress in healthcare applications driven by large language models (LLMs), growing concerns around data privacy, and limited resources; the small language models (SLMs) offer a scalable and clinically viable solution for efficient performance in resource-constrained environments for next-generation healthcare informatics. Our comprehensive survey presents a taxonomic framework to identify and categorize them for healthcare professionals and informaticians. The timeline of healthcare SLM contributions establishes a foundational framework for analyzing models across three dimensions: NLP tasks, stakeholder roles, and the continuum of care. We present a taxonomic framework to identify the architectural foundations for building models from scratch; adapting SLMs to clinical precision through prompting, instruction fine-tuning, and reasoning; and accessibility and sustainability through compression techniques. Our primary objective is to offer a comprehensive survey for healthcare professionals, introducing recent innovations in model optimization and equipping them with curated resources to support future research and development in the field. Aiming to showcase the groundbreaking advancements in SLMs for healthcare, we present a comprehensive compilation of experimental results across widely studied NLP tasks in healthcare to highlight the transformative potential of SLMs in healthcare. The updated repository is available at Github

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 123 of its 234 references that are in the map. First posted on arXiv 23 Apr 2025.

Builds on

123 of 234 references.

References · 123

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2020Longformer: The Long-Document Transformercited by 3,413
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  12. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
38 more
  1. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  4. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  6. 2023LIMA: Less Is More for Alignmentcited by 1,100
  7. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  8. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  9. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  10. 2022A Survey on In-context Learningcited by 721
  11. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  12. 2020The Future of Digital Health with Federated Learningcited by 633
  13. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  14. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  15. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  16. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  17. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  18. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  19. 2021MetaICL: Learning to Learn In Contextcited by 454
  20. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  21. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 364
  22. 2021Explaining Neural Scaling Lawscited by 363
  23. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  24. 2024LoRA Learns Less and Forgets Lesscited by 328
  25. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  26. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
  27. 2023Towards Generalist Biomedical AIcited by 289
  28. 2023Faithful Chain-of-Thought Reasoningcited by 288
  29. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 286
  30. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  31. 2024A Survey on Data Selection for Language Modelscited by 236
  32. 2018Explainable Prediction of Medical Codes from Clinical Textcited by 234
  33. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  34. 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
  35. 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 127
  36. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  37. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  38. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108

50 most cited of 123 All 123 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 13.

18 papers.

Sort:

By area:

  1. 2025Menta: A Small Language Model for On-Device Mental Health Predictioncited by 3
  2. 2026Automatic Prompt Optimization for Dataset-Level Feature Discoverycited by 3
  3. 2026LoopMTP: A looped transformer guided by latent multi-token predictioncited by 3
  4. 2026Reinforcement Learning Improves LLM Accuracy and Reasoning in Disease Classification from Radiology Reportscited by 2
  5. 2025Second Opinion Matters: Towards Adaptive Clinical AI via the Consensus of Expert Model Ensemblecited by 1
  6. 2026DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compressioncited by 1
  7. 2026Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Modelscited by 1
  8. 2025Small Language Models for Emergency Departments Decision Support: A Benchmark Studycited by 0
  9. 2025Dual-stage and Lightweight Patient Chart Summarization for Emergency Physicianscited by 0
  10. 2025Benchmarking and Adapting On-Device LLMs for Clinical Decision Supportcited by 0
  11. 2026Can Small Language Models Handle Context-Summarized Multi-Turn Customer-Service QA? A Synthetic Data-Driven Comparative Evaluationcited by 0
  12. 2026Small Language Models for Privacy-Preserving Clinical Information Extraction in Low-Resource Languagescited by 0
6 more
  1. 2026Resource-Conscious Modeling for Next- Day Discharge Prediction Using Clinical Notescited by 0
  2. 2026Wasserstein Equilibrium Decoding for Reliable Medical Visual Question Answeringcited by 0
  3. 2026TRACE: A taxonomy-grounded synthetic dataset for teaching-program generation and session interpretation in Applied Behavior Analysiscited by 0
  4. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inferencecited by 0
  5. 2026In-Context Learning for Wound Classification with Small Multimodal Language Modelscited by 0
  6. 2026HARISSA: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deploymentcited by 0