mlsanityData to 14 Oct 2026

Federated Self-supervised Learning for Heterogeneous Clients

arXiv 2205.12493 ↗First posted 25 May 2022cs.LG cs.DCcited by 16

Not on the map. Why?

Abstract (arXiv)

Federated Learning has become an important learning paradigm due to its privacy and computational benefits. As the field advances, two key challenges that still remain to be addressed are: (1) system heterogeneity - variability in the compute and/or data resources present on each client, and (2) lack of labeled data in certain federated settings. Several recent developments have tried to overcome these challenges independently. In this work, we propose a unified and systematic framework, Heterogeneous Self-supervised Federated Learning (Hetero-SSFL) for enabling self-supervised learning with federation on heterogeneous clients. The proposed framework allows collaborative representation learning across all the clients without imposing architectural constraints or requiring presence of labeled data. The key idea in Hetero-SSFL is to let each client train its unique self-supervised model and enable the joint learning across clients by aligning the lower dimensional representations on a common dataset. The entire training procedure could be viewed as self and peer-supervised as both the local training and the alignment procedures do not require presence of any labeled data. As in conventional self-supervised learning, the obtained client models are task independent and can be used for varied end-tasks. We provide a convergence guarantee of the proposed framework for non-convex objectives in heterogeneous settings and also empirically demonstrate that our proposed approach outperforms the state of the art methods by a significant margin.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 40 of its 50 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

40 of 50 references.

References · 40

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  8. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  10. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  11. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  12. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
28 more
  1. 2018LEAF: A Benchmark for Federated Settingscited by 792
  2. 2021Model-Contrastive Federated Learningcited by 766
  3. 2017Federated Multi-Task Learningcited by 672
  4. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  5. 2020Federated Learning with Matched Averagingcited by 564
  6. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  7. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  8. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  9. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  10. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  11. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  12. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  13. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 269
  14. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  15. 2019Model Fusion via Optimal Transportcited by 219
  16. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  17. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
  18. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  19. 2020Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learningcited by 112
  20. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  21. 2019ClusterFit: Improving Generalization of Visual Representationscited by 82
  22. 2022Divergence-aware Federated Self-Supervised Learningcited by 73
  23. 2021Collaborative Unsupervised Visual Representation Learning from Decentralized Datacited by 69
  24. 2021Towards Fair Federated Learning with Zero-Shot Data Augmentationcited by 64
  25. 2020Federated Learning via Synthetic Datacited by 54
  26. 2022Architecture Agnostic Federated Learning for Neural Networkscited by 30
  27. 2021SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervisioncited by 27
  28. 2022Federated Learning from Only Unlabeled Data with Class-Conditional-Sharing Clientscited by 20

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 6; 2024: 4; 2025: 4.

16 papers.

Sort:

  1. 2022Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?cited by 12
  2. 2023NeFL: Nested Model Scaling for Federated Learning with System Heterogeneous Clientscited by 7
  3. 2024A Mutual Information Perspective on Federated Contrastive Learningcited by 7
  4. 2023Contrastive encoder pre-training-based clustered federated learning for heterogeneous datacited by 5
  5. 2023Fed-QSSL: A Framework for Personalized Federated Learning under Bitwidth and Data Heterogeneitycited by 4
  6. 2022Federated Training of Dual Encoding Models on Small Non-IID Client Datasetscited by 2
  7. 2023Fast FixMatch: Faster Semi-Supervised Learning with Curriculum Batch Sizecited by 2
  8. 2024EmInspector: Combating Backdoor Attacks in Federated Self-Supervised Learning Through Embedding Inspectioncited by 2
  9. 2025Federated Foundation Model for GI Endoscopy Imagescited by 2
  10. 2025SAFE: Self-Adjustment Federated Learning Framework for Remote Sensing Collaborative Perceptioncited by 1
  11. 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insightscited by 1
  12. 2023Single-round Self-supervised Distributed Learning using Vision Transformercited by 0
4 more
  1. 2023Federated Self-Supervised Learning of Monocular Depth Estimators for Autonomous Vehiclescited by 0
  2. 2024Resource-efficient Layer-wise Federated Self-supervised Learningcited by 0
  3. 2024A deep cut into Split Federated Self-supervised Learningcited by 0
  4. 2025Personalized Federated Heat-Kernel Enhanced Multi-View Clustering via Advanced Tensor Decomposition Techniquescited by 0