# Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision

arXiv: https://arxiv.org/abs/2310.18689 · HTML: https://mlsanity.com/paper/2310.18689/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- Builds on 72 of its 91 references that are in the map.
- First posted on arXiv 28 Oct 2023.

## Builds on (most cited first, 30 of 72)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)

## Built on by (most cited first, 30 of 76)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directions](https://mlsanity.com/paper/2404.03264/) (2024)
- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Models](https://mlsanity.com/paper/2407.00983/) (2024)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [BrainSegFounder: Towards 3D Foundation Models for Neuroimage Segmentation](https://mlsanity.com/paper/2406.10395/) (2024)
- [Loss Functions in the Era of Semantic Segmentation: A Survey and Outlook](https://mlsanity.com/paper/2312.05391/) (2023)
- [FetalCLIP: A Visual-Language Foundation Model for Fetal Ultrasound Image Analysis](https://mlsanity.com/paper/2502.14807/) (2025)
- [Continual Learning in Medical Image Analysis: A Comprehensive Review of Recent Advancements and Future Prospects](https://mlsanity.com/paper/2312.17004/) (2023)
- [Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Survey](https://mlsanity.com/paper/2408.12880/) (2024)
- [A Survey on Trustworthiness in Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2407.15851/) (2024)
- [PromptSmooth: Certifying Robustness of Medical Vision-Language Models via Prompt Learning](https://mlsanity.com/paper/2408.16769/) (2024)
- [PRISM: High-Resolution & Precise Counterfactual Medical Image Generation using Language-guided Stable Diffusion](https://mlsanity.com/paper/2503.00196/) (2025)
- [BAPLe: Backdoor Attacks on Medical Foundational Models using Prompt Learning](https://mlsanity.com/paper/2408.07440/) (2024)
- [VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models](https://mlsanity.com/paper/2503.07575/) (2025)
- [Benchmarking Foundation Models for Zero-Shot Biometric Tasks](https://mlsanity.com/paper/2505.24214/) (2025)
- [Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs](https://mlsanity.com/paper/2502.16435/) (2025)
- [Evaluating Computational Pathology Foundation Models for Prostate Cancer Grading under Distribution Shifts](https://mlsanity.com/paper/2410.06723/) (2024)
- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [RadCLIP: Enhancing Radiologic Image Analysis through Contrastive Language-Image Pre-training](https://mlsanity.com/paper/2403.09948/) (2024)
- [Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review](https://mlsanity.com/paper/2502.14886/) (2025)
- [Pre-training on High Definition X-ray Images: An Experimental Study](https://mlsanity.com/paper/2404.17926/) (2024)
- [Pathology Foundation Models](https://mlsanity.com/paper/2407.21317/) (2024)
- [Masked LoGoNet: Fast and Accurate 3D Image Analysis for Medical Domain](https://mlsanity.com/paper/2402.06190/) (2024)
- [ONCOPILOT: A Promptable CT Foundation Model For Solid Tumor Evaluation](https://mlsanity.com/paper/2410.07908/) (2024)
- [Graph Neural Networks: A suitable Alternative to MLPs in Latent 3D Medical Image Classification?](https://mlsanity.com/paper/2407.17219/) (2024)
- [CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcare](https://mlsanity.com/paper/2603.24157/) (2026)
- [Ethical Framework for Responsible Foundational Models in Medical Imaging](https://mlsanity.com/paper/2406.11868/) (2024)
- [cp_measure: API-first feature extraction for image-based profiling workflows](https://mlsanity.com/paper/2507.01163/) (2025)
- [Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directions](https://mlsanity.com/paper/2511.01284/) (2025)

## Abstract (arXiv)

Foundation models, large-scale, pre-trained deep-learning models adapted to a wide range of downstream tasks have gained significant interest lately in various deep-learning problems undergoing a paradigm shift with the rise of these models. Trained on large-scale dataset to bridge the gap between different modalities, foundation models facilitate contextual reasoning, generalization, and prompt capabilities at test time. The predictions of these models can be adjusted for new tasks by augmenting the model input with task-specific hints called prompts without requiring extensive labeled data and retraining. Capitalizing on the advances in computer vision, medical imaging has also marked a growing interest in these models. To assist researchers in navigating this direction, this survey intends to provide a comprehensive overview of foundation models in the domain of medical imaging. Specifically, we initiate our exploration by providing an exposition of the fundamental concepts forming the basis of foundation models. Subsequently, we offer a methodical taxonomy of foundation models within the medical domain, proposing a classification system primarily structured around training strategies, while also incorporating additional facets such as application domains, imaging modalities, specific organs of interest, and the algorithms integral to these models. Furthermore, we emphasize the practical use case of some selected approaches and then discuss the opportunities, applications, and future directions of these large-scale pre-trained models, for analyzing medical images. In the same vein, we address the prevailing challenges and research pathways associated with foundational models in medical imaging. These encompass the areas of interpretability, data management, computational requirements, and the nuanced issue of contextual comprehension.