mlsanityData to 7 Oct 2026

A Comprehensive Survey of Foundation Models in Medicine

arXiv 2406.10729 ↗First posted 15 Jun 2024cs.LG cs.AI cs.CVcited by 56

Not on the map. Why?

Abstract (arXiv)

Foundation models (FMs) are large-scale deep learning models trained on massive datasets, often using self-supervised learning techniques. These models serve as a versatile base for a wide range of downstream tasks, including those in medicine and healthcare. FMs have demonstrated remarkable success across multiple healthcare domains. However, existing surveys in this field do not comprehensively cover all areas where FMs have made significant strides. In this survey, we present a comprehensive review of FMs in medicine, focusing on their evolution, learning strategies, flagship models, applications, and associated challenges. We examine how prominent FMs, such as the BERT and GPT families, are transforming various aspects of healthcare, including clinical large language models, medical image analysis, and omics research. Additionally, we provide a detailed taxonomy of FM-enabled healthcare applications, spanning clinical natural language processing, medical computer vision, graph learning, and other biology- and omics- related tasks. Despite the transformative potentials of FMs, they also pose unique challenges. This survey delves into these challenges and highlights open research questions and lessons learned to guide researchers and practitioners. Our goal is to provide valuable insights into the capabilities of FMs in health, facilitating responsible deployment and mitigating associated risks.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 249 of its 406 references that are in the map. First posted on arXiv 15 Jun 2024.

Builds on

249 of 406 references.

References · 249

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  8. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  9. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  10. 2023GPT-4 Technical Reportcited by 18,132
  11. 2022Training language models to follow instructions with human feedbackcited by 15,712
  12. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
38 more
  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2023Segment Anythingcited by 8,158
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2020Scaling Laws for Neural Language Modelscited by 6,411
  13. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  14. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  15. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  16. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  17. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  18. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  19. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  20. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  21. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  22. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  23. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  24. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  25. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  26. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  27. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  28. 2020Learning to summarize from human feedbackcited by 2,562
  29. 2023A Survey of Large Language Modelscited by 2,537
  30. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  31. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  32. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  33. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  34. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,869
  35. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  36. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  37. 2020GLU Variants Improve Transformercited by 1,724
  38. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722

50 most cited of 249 All 249 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 28; 2026: 26.

56 papers.

Sort:

By area:

  1. 2025MedSAM2: Segment Anything in 3D Medical Images and Videoscited by 88
  2. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  3. 2025Benchmarking Foundation Models for Zero-Shot Biometric Taskscited by 13
  4. 2025On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?cited by 10
  5. 2025Representation Potentials of Foundation Models for Multimodal Alignment: A Surveycited by 10
  6. 2025Vision Foundation Models in Medical Image Analysis: Advances and Challengescited by 9
  7. 2025Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Reviewcited by 8
  8. 2025Med-GLIP: Advancing Medical Language-Image Pre-training with Large-scale Grounded Datasetcited by 7
  9. 2025Prompt to Polyp: Medical Text-Conditioned Image Synthesis with Diffusion Modelscited by 5
  10. 2025The Road to Generalizable Neuro-Symbolic Learning Should be Paved with Foundation Modelscited by 3
  11. 2025Benchmarking Foundation Models and Parameter-Efficient Fine-Tuning for Prognosis Prediction in Medical Imagingcited by 3
  12. 2025Impact of Clinical Image Quality on Efficient Foundation Model Finetuningcited by 3
38 more
  1. 2025SingLEM: Single-Channel Large EEG Modelcited by 3
  2. 2025Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directionscited by 3
  3. 2026Causal Data Augmentation for Robust Fine-Tuning of Tabular Foundation Modelscited by 3
  4. 2026MARCH: Multi-Agent Radiology Clinical Hierarchy for CT Report Generationcited by 3
  5. 2026Medical world models in healthcare: foundations, applications, and challenges for trustworthy clinical translationcited by 3
  6. 2025Generalist Models in Medical Image Segmentation: A Survey and Performance Comparison with Task-Specific Approachescited by 2
  7. 2025Benchmarking Foundation Models for Mitotic Figure Classificationcited by 2
  8. 2026A data- and compute-efficient chest X-ray foundation model beyond aggressive scalingcited by 2
  9. 2026Retrieval-aligned Tabular Foundation Models Enable Robust Clinical Risk Prediction in Electronic Health Records Under Real-world Constraintscited by 2
  10. 2024HistoEncoder: a digital pathology foundation model for prostate cancercited by 1
  11. 2025SAM2-UNeXT: An Improved High-Resolution Baseline for Adapting Foundation Models to Downstream Segmentation Taskscited by 1
  12. 2025FoundBioNet: A Foundation-Based Model for IDH Genotyping of Glioma from Multi-Parametric MRIcited by 1
  13. 2025Machine Learning for Medicine Must Be Interpretable, Shareable, Reproducible and Accountable by Designcited by 1
  14. 2026Breaking Semantic Hegemony: Decoupling Principal and Residual Subspaces for Generalized OOD Detectioncited by 1
  15. 2026MeFEm: Medical Face Embedding modelcited by 1
  16. 2026Seeing Like Radiologists: Context- and Gaze-Guided Vision-Language Pretraining for Chest X-rayscited by 1
  17. 2026Foundation Model for Cardiac Time Series via Masked Latent Attentioncited by 1
  18. 2026GenMed: A Pairwise Generative Reformulation of Medical Diagnostic Taskscited by 1
  19. 2024Cancer-Answer: Empowering Cancer Care with Advanced Large Language Modelscited by 0
  20. 2025Towards a Multimodal MRI-Based Foundation Model for Multi-Level Feature Exploration in Segmentation, Molecular Subtyping, and Grading of Gliomacited by 0
  21. 2025Multi-Task Learning for Extracting Menstrual Characteristics from Clinical Notescited by 0
  22. 2025SAMba-UNet: SAM2-Mamba UNet for Cardiac MRI in Medical Robotic Perceptioncited by 0
  23. 2025Prompt Mechanisms in Medical Imaging: A Comprehensive Surveycited by 0
  24. 2025Leveraging Generic Foundation Models for Multimodal Surgical Data Analysiscited by 0
  25. 2025CLAPS: A CLIP-Unified Auto-Prompt Segmentation for Multi-Modal Retinal Imagingcited by 0
  26. 2025ALL-PET: A Low-resource and Low-shot PET Foundation Model in Projection Domaincited by 0
  27. 2025Towards Self-Supervised Foundation Models for Critical Care Time Seriescited by 0
  28. 2025Structured Output Regularization: a framework for few-shot transfer learningcited by 0
  29. 2026Multi-scale Graph Autoregressive Modeling: Molecular Property Prediction via Next Token Predictioncited by 0
  30. 2026Specializing Foundation Models via Mixture of Low-Rank Experts for Comprehensive Head CT Analysiscited by 0
  31. 2026GP-VM×SMA: Benchmarking General-Purpose Vision Models and Specialized Architectures for 2D Medical Image Segmentationcited by 0
  32. 2026Toward Clinically Ready Foundation Models in Medical Image Analysis: Adaptation Mechanisms and Deployment Trade-offscited by 0
  33. 2026Weakly Supervised Multicenter Nancy Index Scoring in Ulcerative Colitis Using Foundation Modelscited by 0
  34. 2026Cross-Stage Attention Multi-Expert Network for Radiologist-Inspired Breast Ultrasound Diagnosiscited by 0
  35. 2026How Hard Can It Be? Hardness-Aware Multi-Objective Unlearningcited by 0
  36. 2026Beyond Visual Cues: CoT-Enhanced Reasoning for Semi-supervised Medical Image Segmentationcited by 0
  37. 2026Pulmonary Embolism Risk Stratification from CTPA and Medical Records: Vascular Graphs Are Not All You Needcited by 0
  38. 2026SonoCLIP: Mask-Guided Region-Aware Vision-Language Pretraining for Fetal Ultrasound Analysiscited by 0

50 most cited of 56 All 56 (JSON)