# Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imaging

arXiv: https://arxiv.org/abs/2205.08576 · HTML: https://mlsanity.com/paper/2205.08576/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 77 references that are in the map.
- First posted on arXiv 17 May 2022.

## Builds on (most cited first, 30 of 51)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)

## Built on by (most cited first, 30 of 39)

- [Selective Aggregation for Low-Rank Adaptation in Federated Learning](https://mlsanity.com/paper/2410.01463/) (2024)
- [Federated Learning for Medical Image Analysis: A Survey](https://mlsanity.com/paper/2306.05980/) (2023)
- [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](https://mlsanity.com/paper/2301.03505/) (2023)
- [Federated Learning-Empowered AI-Generated Content in Wireless Networks](https://mlsanity.com/paper/2307.07146/) (2023)
- [Federated Learning for Computer Vision](https://mlsanity.com/paper/2308.13558/) (2023)
- [Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directions](https://mlsanity.com/paper/2411.12377/) (2024)
- [One-shot Federated Learning via Synthetic Distiller-Distillate Communication](https://mlsanity.com/paper/2412.05186/) (2024)
- [Mobility-Aware Federated Self-supervised Learning in Vehicular Network](https://mlsanity.com/paper/2408.00256/) (2024)
- [Tackling Data Heterogeneity in Federated Learning via Loss Decomposition](https://mlsanity.com/paper/2408.12300/) (2024)
- [Exploring the Vulnerabilities of Federated Learning: A Deep Dive into Gradient Inversion Attacks](https://mlsanity.com/paper/2503.11514/) (2025)
- [DRL-Based Resource Allocation for Motion Blur Resistant Federated Self-Supervised Learning in IoV](https://mlsanity.com/paper/2408.09194/) (2024)
- [FedLPPA: Learning Personalized Prompt and Aggregation for Federated Weakly-supervised Medical Image Segmentation](https://mlsanity.com/paper/2402.17502/) (2024)
- [Federated Learning for Medical Image Classification: A Comprehensive Benchmark](https://mlsanity.com/paper/2504.05238/) (2025)
- [Boosting multi-demographic federated learning for chest radiograph analysis using general-purpose self-supervised representations](https://mlsanity.com/paper/2504.08584/) (2025)
- [Feasibility of Federated Learning from Client Databases with Different Brain Diseases and MRI Modalities](https://mlsanity.com/paper/2406.11636/) (2024)
- [Navigating Distribution Shifts in Medical Image Analysis: A Survey](https://mlsanity.com/paper/2411.05824/) (2024)
- [Privacy-preserving machine learning for healthcare: open challenges and future perspectives](https://mlsanity.com/paper/2303.15563/) (2023)
- [SelfFed: Self-Supervised Federated Learning for Data Heterogeneity and Label Scarcity in Medical Images](https://mlsanity.com/paper/2307.01514/) (2023)
- [FedDBL: Communication and Data Efficient Federated Deep-Broad Learning for Histopathological Tissue Classification](https://mlsanity.com/paper/2302.12662/) (2023)
- [Take Your Pick: Enabling Effective Personalized Federated Learning within Low-dimensional Feature Space](https://mlsanity.com/paper/2307.13995/) (2023)
- [FedConv: Enhancing Convolutional Neural Networks for Handling Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2310.04412/) (2023)
- [The Effects of Data Imbalance Under a Federated Learning Approach for Credit Risk Forecasting](https://mlsanity.com/paper/2401.07234/) (2024)
- [Automatic Prediction of Stroke Treatment Outcomes: Latest Advances and Perspectives](https://mlsanity.com/paper/2412.04812/) (2024)
- [Technical and Legal Aspects of Federated Learning in Bioinformatics: Applications, Challenges and Opportunities](https://mlsanity.com/paper/2503.09649/) (2025)
- [Federated Foundation Model for GI Endoscopy Images](https://mlsanity.com/paper/2505.24108/) (2025)
- [Balancing Privacy and Performance for Private Federated Learning Algorithms](https://mlsanity.com/paper/2304.05127/) (2023)
- [FGAD: Self-boosted Knowledge Distillation for An Effective Federated Graph Anomaly Detection Framework](https://mlsanity.com/paper/2402.12761/) (2024)
- [Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity](https://mlsanity.com/paper/2404.03854/) (2024)
- [Towards Privacy-Preserving Medical Imaging: Federated Learning with Differential Privacy and Secure Aggregation Using a Modified ResNet Architecture](https://mlsanity.com/paper/2412.00687/) (2024)
- [Task-Specific Knowledge Distillation from the Vision Foundation Model for Enhanced Medical Image Segmentation](https://mlsanity.com/paper/2503.06976/) (2025)

## Abstract (arXiv)

The collection and curation of large-scale medical datasets from multiple institutions is essential for training accurate deep learning models, but privacy concerns often hinder data sharing. Federated learning (FL) is a promising solution that enables privacy-preserving collaborative learning among different institutions, but it generally suffers from performance deterioration due to heterogeneous data distributions and a lack of quality labeled data. In this paper, we present a robust and label-efficient self-supervised FL framework for medical image analysis. Our method introduces a novel Transformer-based self-supervised pre-training paradigm that pre-trains models directly on decentralized target task datasets using masked image modeling, to facilitate more robust representation learning on heterogeneous data and effective knowledge transfer to downstream models. Extensive empirical results on simulated and real-world medical imaging non-IID federated datasets show that masked image modeling with Transformers significantly improves the robustness of models against various degrees of data heterogeneity. Notably, under severe data heterogeneity, our method, without relying on any additional pre-training data, achieves an improvement of 5.06%, 1.53% and 4.58% in test accuracy on retinal, dermatology and chest X-ray classification compared to the supervised baseline with ImageNet pre-training. In addition, we show that our federated self-supervised pre-training methods yield models that generalize better to out-of-distribution data and perform more effectively when fine-tuning with limited labeled data, compared to existing FL algorithms. The code is available at https://github.com/rui-yan/SSL-FL.