# FedFMS: Exploring Federated Foundation Models for Medical Image Segmentation

arXiv: https://arxiv.org/abs/2403.05408 · HTML: https://mlsanity.com/paper/2403.05408/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 8 Mar 2024.

## Builds on (most cited first, 15 of 15)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space](https://mlsanity.com/paper/2103.06030/) (2021)
- [REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs](https://mlsanity.com/paper/1910.03667/) (2019)
- [Privacy-preserving Federated Brain Tumour Segmentation](https://mlsanity.com/paper/1910.00962/) (2019)
- [Medical Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2009.13120/) (2020)
- [PanNuke Dataset Extension, Insights and Baselines](https://mlsanity.com/paper/2003.10778/) (2020)
- [The Federated Tumor Segmentation (FeTS) Challenge](https://mlsanity.com/paper/2105.05874/) (2021)
- [Federated Learning for Medical Applications: A Taxonomy, Current Trends, Challenges, and Future Research Directions](https://mlsanity.com/paper/2208.03392/) (2022)
- [G1020: A Benchmark Retinal Fundus Image Dataset for Computer-Aided Glaucoma Detection](https://mlsanity.com/paper/2006.09158/) (2020)

## Built on by (most cited first, 21 of 21)

- [Ten Challenging Problems in Federated Foundation Models](https://mlsanity.com/paper/2502.12176/) (2025)
- [A Survey on Trustworthiness in Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2407.15851/) (2024)
- [FedSynthCT-Brain: A Federated Learning Framework for Multi-Institutional Brain MRI-to-CT Synthesis](https://mlsanity.com/paper/2412.06690/) (2024)
- [Federated nnU-Net for Privacy-Preserving Medical Image Segmentation](https://mlsanity.com/paper/2503.02549/) (2025)
- [Federated Client-tailored Adapter for Medical Image Segmentation](https://mlsanity.com/paper/2504.18020/) (2025)
- [Equitable Federated Learning with NCA](https://mlsanity.com/paper/2506.21735/) (2025)
- [FedEFM: Federated Endovascular Foundation Model with Unseen Data](https://mlsanity.com/paper/2501.16992/) (2025)
- [Federated Foundation Model for GI Endoscopy Images](https://mlsanity.com/paper/2505.24108/) (2025)
- [FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Models](https://mlsanity.com/paper/2508.03356/) (2025)
- [A New One-Shot Federated Learning Framework for Medical Imaging Classification with Feature-Guided Rectified Flow and Knowledge Distillation](https://mlsanity.com/paper/2507.19045/) (2025)
- [Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights](https://mlsanity.com/paper/2509.05142/) (2025)
- [pFedSAM: Personalized Federated Learning of Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2509.15638/) (2025)
- [OmniFM: Toward Modality-Robust and Task-Agnostic Federated Learning for Heterogeneous Medical Imaging](https://mlsanity.com/paper/2603.21660/) (2026)
- [FedSCA: Federated Tuning with Similarity-guided Collaborative Aggregation for Heterogeneous Medical Image Segmentation](https://mlsanity.com/paper/2503.15390/) (2025)
- [Brain Imaging Foundation Models, Are We There Yet? A Systematic Review of Foundation Models for Brain Imaging and Biomedical Research](https://mlsanity.com/paper/2506.13306/) (2025)
- [FRIEREN: Federated Learning with Vision-Language Regularization for Segmentation](https://mlsanity.com/paper/2510.02114/) (2025)
- [FedEU: Evidential Uncertainty-Driven Federated Fine-Tuning of Vision Foundation Models for Remote Sensing Image Segmentation](https://mlsanity.com/paper/2603.07468/) (2026)
- [Shift-Dependent Asymmetry: Orthogonal Inverse Low-Rank Adaptation for Federated Medical Segmentation](https://mlsanity.com/paper/2606.08687/) (2026)
- [EFFEKT: Efficient Federated Knowledge Transfer to Foundation Models](https://mlsanity.com/paper/2608.08138/) (2026)
- [CoM3eT: A foundation model for medical image analysis through federated, multidimensional context integration](https://mlsanity.com/paper/2608.16268/) (2026)
- [Federated LoRA Adaptation of BiomedCLIP Across Four International Chest X-Ray Cohorts](https://mlsanity.com/paper/2609.02101/) (2026)

## Abstract (arXiv)

Medical image segmentation is crucial for clinical diagnosis. The Segmentation Anything Model (SAM) serves as a powerful foundation model for visual segmentation and can be adapted for medical image segmentation. However, medical imaging data typically contain privacy-sensitive information, making it challenging to train foundation models with centralized storage and sharing. To date, there are few foundation models tailored for medical image deployment within the federated learning framework, and the segmentation performance, as well as the efficiency of communication and training, remain unexplored. In response to these issues, we developed Federated Foundation models for Medical image Segmentation (FedFMS), which includes the Federated SAM (FedSAM) and a communication and training-efficient Federated SAM with Medical SAM Adapter (FedMSA). Comprehensive experiments on diverse datasets are conducted to investigate the performance disparities between centralized training and federated learning across various configurations of FedFMS. The experiments revealed that FedFMS could achieve performance comparable to models trained via centralized training methods while maintaining privacy. Furthermore, FedMSA demonstrated the potential to enhance communication and training efficiency. Our model implementation codes are available at https://github.com/LIU-YUXI/FedFMS.