# Federated Self-supervised Learning for Heterogeneous Clients

arXiv: https://arxiv.org/abs/2205.12493 · HTML: https://mlsanity.com/paper/2205.12493/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 50 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 30 of 40)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Ditto: Fair and Robust Federated Learning Through Personalization](https://mlsanity.com/paper/2012.04221/) (2020)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [FedProto: Federated Prototype Learning across Heterogeneous Clients](https://mlsanity.com/paper/2105.00243/) (2021)
- [Personalized Federated Learning: A Meta-Learning Approach](https://mlsanity.com/paper/2002.07948/) (2020)
- [Improving Federated Learning Personalization via Model Agnostic Meta Learning](https://mlsanity.com/paper/1909.12488/) (2019)
- [No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2106.05001/) (2021)
- [Model Fusion via Optimal Transport](https://mlsanity.com/paper/1910.05653/) (2019)
- [FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning](https://mlsanity.com/paper/2009.01974/) (2020)
- [FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data](https://mlsanity.com/paper/2005.11418/) (2020)
- [Salvaging Federated Learning by Local Adaptation](https://mlsanity.com/paper/2002.04758/) (2020)

## Built on by (most cited first, 16 of 16)

- [Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?](https://mlsanity.com/paper/2210.10947/) (2022)
- [NeFL: Nested Model Scaling for Federated Learning with System Heterogeneous Clients](https://mlsanity.com/paper/2308.07761/) (2023)
- [A Mutual Information Perspective on Federated Contrastive Learning](https://mlsanity.com/paper/2405.02081/) (2024)
- [Contrastive encoder pre-training-based clustered federated learning for heterogeneous data](https://mlsanity.com/paper/2311.16535/) (2023)
- [Fed-QSSL: A Framework for Personalized Federated Learning under Bitwidth and Data Heterogeneity](https://mlsanity.com/paper/2312.13380/) (2023)
- [Federated Training of Dual Encoding Models on Small Non-IID Client Datasets](https://mlsanity.com/paper/2210.00092/) (2022)
- [Fast FixMatch: Faster Semi-Supervised Learning with Curriculum Batch Size](https://mlsanity.com/paper/2309.03469/) (2023)
- [EmInspector: Combating Backdoor Attacks in Federated Self-Supervised Learning Through Embedding Inspection](https://mlsanity.com/paper/2405.13080/) (2024)
- [Federated Foundation Model for GI Endoscopy Images](https://mlsanity.com/paper/2505.24108/) (2025)
- [SAFE: Self-Adjustment Federated Learning Framework for Remote Sensing Collaborative Perception](https://mlsanity.com/paper/2504.03700/) (2025)
- [Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights](https://mlsanity.com/paper/2509.05142/) (2025)
- [Single-round Self-supervised Distributed Learning using Vision Transformer](https://mlsanity.com/paper/2301.02064/) (2023)
- [Federated Self-Supervised Learning of Monocular Depth Estimators for Autonomous Vehicles](https://mlsanity.com/paper/2310.04837/) (2023)
- [Resource-efficient Layer-wise Federated Self-supervised Learning](https://mlsanity.com/paper/2401.11647/) (2024)
- [A deep cut into Split Federated Self-supervised Learning](https://mlsanity.com/paper/2406.08267/) (2024)
- [Personalized Federated Heat-Kernel Enhanced Multi-View Clustering via Advanced Tensor Decomposition Techniques](https://mlsanity.com/paper/2509.16101/) (2025)

## Abstract (arXiv)

Federated Learning has become an important learning paradigm due to its privacy and computational benefits. As the field advances, two key challenges that still remain to be addressed are: (1) system heterogeneity - variability in the compute and/or data resources present on each client, and (2) lack of labeled data in certain federated settings. Several recent developments have tried to overcome these challenges independently. In this work, we propose a unified and systematic framework, Heterogeneous Self-supervised Federated Learning (Hetero-SSFL) for enabling self-supervised learning with federation on heterogeneous clients. The proposed framework allows collaborative representation learning across all the clients without imposing architectural constraints or requiring presence of labeled data. The key idea in Hetero-SSFL is to let each client train its unique self-supervised model and enable the joint learning across clients by aligning the lower dimensional representations on a common dataset. The entire training procedure could be viewed as self and peer-supervised as both the local training and the alignment procedures do not require presence of any labeled data. As in conventional self-supervised learning, the obtained client models are task independent and can be used for varied end-tasks. We provide a convergence guarantee of the proposed framework for non-convex objectives in heterogeneous settings and also empirically demonstrate that our proposed approach outperforms the state of the art methods by a significant margin.