# Foundation Models in Radiology: What, How, When, Why and Why Not

arXiv: https://arxiv.org/abs/2411.18730 · HTML: https://mlsanity.com/paper/2411.18730/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 56 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Builds on (most cited first, 13 of 13)

- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Towards Generalist Biomedical AI](https://mlsanity.com/paper/2307.14334/) (2023)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [Segment Anything in Medical Images](https://mlsanity.com/paper/2304.12306/) (2023)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards](https://mlsanity.com/paper/2210.12186/) (2022)
- [MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records](https://mlsanity.com/paper/2308.14089/) (2023)
- [Translating Radiology Reports into Plain Language using ChatGPT and GPT-4 with Prompt Learning: Promising Results, Limitations, and Potential](https://mlsanity.com/paper/2303.09038/) (2023)
- [OPTIMAM Mammography Image Database: a large scale resource of mammography images and clinical data](https://mlsanity.com/paper/2004.04742/) (2020)
- [Risk of Bias in Chest Radiography Deep Learning Foundation Models](https://mlsanity.com/paper/2209.02965/) (2022)
- [GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Models](https://mlsanity.com/paper/2312.06315/) (2023)

## Built on by (most cited first, 28 of 28)

- [SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence](https://mlsanity.com/paper/2506.02555/) (2025)
- [Fr'echet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasets](https://mlsanity.com/paper/2412.01496/) (2024)
- [Curia: A Multi-Modal Foundation Model for Radiology](https://mlsanity.com/paper/2509.06830/) (2025)
- [From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine](https://mlsanity.com/paper/2502.09242/) (2025)
- [Agentic Systems in Radiology: Design, Applications, Evaluation, and Challenges](https://mlsanity.com/paper/2510.09404/) (2025)
- [Bias and Generalizability of Foundation Models across Datasets in Breast Mammography](https://mlsanity.com/paper/2505.10579/) (2025)
- [Render-FM: Feedforward Model for Real-time Photorealistic Volumetric Rendering](https://mlsanity.com/paper/2505.17338/) (2025)
- [From Embeddings to Accuracy: Comparing Foundation Models for Radiographic Classification](https://mlsanity.com/paper/2505.10823/) (2025)
- [Enabling Ultra-Fast Cardiovascular Imaging Across Heterogeneous Clinical Environments with A Generalist Foundation Model and Multimodal Database](https://mlsanity.com/paper/2512.21652/) (2025)
- [GreenRFM: Learning a resource-efficient radiology vision-language foundation model via supervision-centric pre-training](https://mlsanity.com/paper/2603.06467/) (2026)
- [Surg-R1: A Hierarchical Reasoning Foundation Model for Scalable and Interpretable Surgical Decision Support with Multi-Center Clinical Validation](https://mlsanity.com/paper/2603.12430/) (2026)
- [Toward a robust lesion detection model in breast DCE-MRI: adapting foundation models to high-risk women](https://mlsanity.com/paper/2509.02710/) (2025)
- [Process Reward Models for Sentence-Level Verification of LVLM Radiology Reports](https://mlsanity.com/paper/2510.23217/) (2025)
- [TomoGraphView: 3D Medical Image Classification with Omnidirectional Slice Representations and Graph Neural Networks](https://mlsanity.com/paper/2511.09605/) (2025)
- [A Hierarchical Benchmark of Foundation Models for Dermatology](https://mlsanity.com/paper/2601.12382/) (2026)
- [A texture-based framework for foundational ultrasound models](https://mlsanity.com/paper/2602.01444/) (2026)
- [Vision Foundation Models in Radiology: A Scoping Review of Data, Methodology, Evaluation and Clinical Translation](https://mlsanity.com/paper/2607.07219/) (2026)
- [Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift](https://mlsanity.com/paper/2607.10358/) (2026)
- [Do Medical Foundation Models Generalize on the African Brain?](https://mlsanity.com/paper/2607.28771/) (2026)
- [Consensus in the Parliament of AI: Harmonized Multi-Region CT-Radiomics and Foundation-Model Signatures for Multicentre NSCLC Risk Stratification](https://mlsanity.com/paper/2505.17893/) (2025)
- [NeuroRAD-FM: A Foundation Model for Neuro-Oncology with Distributionally Robust Training](https://mlsanity.com/paper/2509.15416/) (2025)
- [Adapted Foundation Models for Breast MRI Triaging in Contrast-Enhanced and Non-Contrast Enhanced Protocols](https://mlsanity.com/paper/2511.05967/) (2025)
- [Evaluating the Clinical Impact of Generative Inpainting on Bone Age Estimation](https://mlsanity.com/paper/2511.23066/) (2025)
- [Tractogram foundation model](https://mlsanity.com/paper/2606.09893/) (2026)
- [Learning from Acquisition: Metadata-driven Multimodal Pre-training for Cardiac MRI](https://mlsanity.com/paper/2606.28991/) (2026)
- [Development of an automated, reliable, and clinically meaningful artificial intelligence (AI) tool for diagnosing cardiac disease from conventional cardiovascular magnetic resonance (CMR) images](https://mlsanity.com/paper/2607.20087/) (2026)
- [Image-Based Breast Implant Detection for Mammography Dataset Curation and Near-Real-Time Deployment: Comparing Foundation Models and Task-Specific Convolutional Models](https://mlsanity.com/paper/2610.03817/) (2026)
- [Beyond the Encoder: Object-Centric Representations Emerging from Segmentation Foundation Models](https://mlsanity.com/paper/2610.18543/) (2026)

## Abstract (arXiv)

Recent advances in artificial intelligence have witnessed the emergence of large-scale deep learning models capable of interpreting and generating both textual and imaging data. Such models, typically referred to as foundation models, are trained on extensive corpora of unlabeled data and demonstrate high performance across various tasks. Foundation models have recently received extensive attention from academic, industry, and regulatory bodies. Given the potentially transformative impact that foundation models can have on the field of radiology, this review aims to establish a standardized terminology concerning foundation models, with a specific focus on the requirements of training data, model training paradigms, model capabilities, and evaluation strategies. We further outline potential pathways to facilitate the training of radiology-specific foundation models, with a critical emphasis on elucidating both the benefits and challenges associated with such models. Overall, we envision that this review can unify technical advances and clinical needs in the training of foundation models for radiology in a safe and responsible manner, for ultimately benefiting patients, providers, and radiologists.