# The Rise of Small Language Models in Healthcare: A Comprehensive Survey

arXiv: https://arxiv.org/abs/2504.17119 · HTML: https://mlsanity.com/paper/2504.17119/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 123 of its 234 references that are in the map.
- First posted on arXiv 23 Apr 2025.

## Builds on (most cited first, 30 of 123)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)

## Built on by (most cited first, 18 of 18)

- [Menta: A Small Language Model for On-Device Mental Health Prediction](https://mlsanity.com/paper/2512.02716/) (2025)
- [Automatic Prompt Optimization for Dataset-Level Feature Discovery](https://mlsanity.com/paper/2601.13922/) (2026)
- [LoopMTP: A looped transformer guided by latent multi-token prediction](https://mlsanity.com/paper/2608.03624/) (2026)
- [Reinforcement Learning Improves LLM Accuracy and Reasoning in Disease Classification from Radiology Reports](https://mlsanity.com/paper/2604.19060/) (2026)
- [Second Opinion Matters: Towards Adaptive Clinical AI via the Consensus of Expert Model Ensemble](https://mlsanity.com/paper/2505.23075/) (2025)
- [DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression](https://mlsanity.com/paper/2607.06523/) (2026)
- [Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Models](https://mlsanity.com/paper/2608.17183/) (2026)
- [Small Language Models for Emergency Departments Decision Support: A Benchmark Study](https://mlsanity.com/paper/2510.04032/) (2025)
- [Dual-stage and Lightweight Patient Chart Summarization for Emergency Physicians](https://mlsanity.com/paper/2510.06263/) (2025)
- [Benchmarking and Adapting On-Device LLMs for Clinical Decision Support](https://mlsanity.com/paper/2601.03266/) (2025)
- [Can Small Language Models Handle Context-Summarized Multi-Turn Customer-Service QA? A Synthetic Data-Driven Comparative Evaluation](https://mlsanity.com/paper/2602.00665/) (2026)
- [Small Language Models for Privacy-Preserving Clinical Information Extraction in Low-Resource Languages](https://mlsanity.com/paper/2602.21374/) (2026)
- [Resource-Conscious Modeling for Next- Day Discharge Prediction Using Clinical Notes](https://mlsanity.com/paper/2604.03498/) (2026)
- [Wasserstein Equilibrium Decoding for Reliable Medical Visual Question Answering](https://mlsanity.com/paper/2605.18313/) (2026)
- [TRACE: A taxonomy-grounded synthetic dataset for teaching-program generation and session interpretation in Applied Behavior Analysis](https://mlsanity.com/paper/2605.25038/) (2026)
- [FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference](https://mlsanity.com/paper/2607.06519/) (2026)
- [In-Context Learning for Wound Classification with Small Multimodal Language Models](https://mlsanity.com/paper/2607.18819/) (2026)
- [HARISSA: Inference-Time Self-Checks for Efficient and Safe Local Language Model Deployment](https://mlsanity.com/paper/2609.38006/) (2026)

## Abstract (arXiv)

Despite substantial progress in healthcare applications driven by large language models (LLMs), growing concerns around data privacy, and limited resources; the small language models (SLMs) offer a scalable and clinically viable solution for efficient performance in resource-constrained environments for next-generation healthcare informatics. Our comprehensive survey presents a taxonomic framework to identify and categorize them for healthcare professionals and informaticians. The timeline of healthcare SLM contributions establishes a foundational framework for analyzing models across three dimensions: NLP tasks, stakeholder roles, and the continuum of care. We present a taxonomic framework to identify the architectural foundations for building models from scratch; adapting SLMs to clinical precision through prompting, instruction fine-tuning, and reasoning; and accessibility and sustainability through compression techniques. Our primary objective is to offer a comprehensive survey for healthcare professionals, introducing recent innovations in model optimization and equipping them with curated resources to support future research and development in the field. Aiming to showcase the groundbreaking advancements in SLMs for healthcare, we present a comprehensive compilation of experimental results across widely studied NLP tasks in healthcare to highlight the transformative potential of SLMs in healthcare. The updated repository is available at Github