Divergence-aware Federated Self-Supervised Learning
Not on the map. Why?
Abstract (arXiv)
Self-supervised learning (SSL) is capable of learning remarkable representations from centrally available data. Recent works further implement federated learning with SSL to learn from rapidly growing decentralized unlabeled images (e.g., from cameras and phones), often resulted from privacy constraints. Extensive attention has been paid to SSL approaches based on Siamese networks. However, such an effort has not yet revealed deep insights into various fundamental building blocks for the federated self-supervised learning (FedSSL) architecture. We aim to fill in this gap via in-depth empirical study and propose a new method to tackle the non-independently and identically distributed (non-IID) data problem of decentralized data. Firstly, we introduce a generalized FedSSL framework that embraces existing SSL methods based on Siamese networks and presents flexibility catering to future methods. In this framework, a server coordinates multiple clients to conduct SSL training and periodically updates local models of clients with the aggregated global model. Using the framework, our study uncovers unique insights of FedSSL: 1) stop-gradient operation, previously reported to be essential, is not always necessary in FedSSL; 2) retaining local knowledge of clients in FedSSL is particularly beneficial for non-IID data. Inspired by the insights, we then propose a new approach for model update, Federated Divergence-aware Exponential Moving Average update (FedEMA). FedEMA updates local models of clients adaptively using EMA of the global model, where the decay rate is dynamically measured by model divergence. Extensive experiments demonstrate that FedEMA outperforms existing methods by 3-4% on linear evaluation. We hope that this work will provide useful insights for future research.
Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 41 of its 48 references that are in the map. First posted on arXiv 9 Apr 2022.
Builds on
41 of 48 references.
References · 41
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2020Exploring Simple Siamese Representation Learningcited by 2,853
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
- 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
29 more
- 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
- 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
- 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
- 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
- 2016Colorful Image Colorizationcited by 1,113
- 2018Federated Learning with Non-IID Datacited by 1,099
- 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
- 2018LEAF: A Benchmark for Federated Settingscited by 780
- 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
- 2020Federated Learning with Matched Averagingcited by 564
- 2021Towards Personalized Federated Learningcited by 462
- 2020Adaptive Personalized Federated Learningcited by 353
- 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
- 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
- 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
- 2019Revisiting Self-Supervised Visual Representation Learningcited by 250
- 2019Federated Adversarial Domain Adaptationcited by 140
- 2020Lower Bounds and Optimal Algorithms for Personalized Federated Learningcited by 122
- 2020Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learningcited by 112
- 2020Federated Unsupervised Representation Learningcited by 92
- 2021Collaborative Unsupervised Visual Representation Learning from Decentralized Datacited by 68
- 2020Performance Optimization for Federated Person Re-identification via Benchmark Analysiscited by 50
- 2019Real-World Image Datasets for Federated Learningcited by 30
- 2020Neural Data Server: A Large-Scale Search Engine for Transfer Learning Datacited by 27
- 2020Towards Utilizing Unlabeled Data in Federated Learning: A Survey and Prospectivecited by 27
- 2021EasyFL: A Low-code Federated Learning Platform For Dummiescited by 20
- 2021Joint Optimization in Edge-Cloud Continuum for Federated Unsupervised Person Re-identificationcited by 20
- 2020Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Modelscited by 10
- 2021Towards Unsupervised Domain Adaptation for Deep Face Recognition under Privacy Constraints via Federated Learningcited by 10
Built on by
Citing papers per year: 2022: 17; 2023: 21; 2024: 18; 2025: 12; 2026: 3.
71 papers.
Sort:
By area:
- 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 83
- 2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
- 2022Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imagingcited by 39
- 2023Personalized Federated Learning under Mixture of Distributionscited by 37
- 2024Relaxed Contrastive Learning for Federated Learningcited by 32
- 2023L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learningcited by 29
- 2023Federated Learning for Computer Visioncited by 22
- 2022A Hybrid Self-Supervised Learning Framework for Vertical Federated Learningcited by 17
- 2022Federated Self-supervised Learning for Heterogeneous Clientscited by 16
- 2022Distributed Contrastive Learning for Medical Image Segmentationcited by 16
- 2024FedMef: Towards Memory-efficient Federated Dynamic Pruningcited by 14
- 2023Re-thinking Federated Active Learning based on Inter-class Diversitycited by 13
38 more
- 2023MergeSFL: Split Federated Learning with Feature Merging and Batch Size Regulationcited by 13
- 2022Autonomy and Intelligence in the Computing Continuum: Challenges, Enablers, and Future Directions for Orchestrationcited by 12
- 2022Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?cited by 12
- 2023Federated Learning without Full Labels: A Surveycited by 12
- 2024Rethinking the Representation in Federated Unsupervised Learning with Non-IID Datacited by 12
- 2023FedWon: Triumphing Multi-domain Federated Learning Without Normalizationcited by 11
- 2022Optimizing Performance of Federated Person Re-identification: Benchmarking and Analysiscited by 9
- 2023Combating Data Imbalances in Federated Semi-supervised Learning with Dual Regulatorscited by 9
- 2023MAS: Towards Resource-Efficient Federated Multiple-Task Learningcited by 9
- 2023Federated Learning for Large-Scale Scene Modeling with Neural Radiance Fieldscited by 8
- 2023NeFL: Nested Model Scaling for Federated Learning with System Heterogeneous Clientscited by 7
- 2024A Mutual Information Perspective on Federated Contrastive Learningcited by 7
- 2024Federated Unsupervised Domain Generalization using Global and Local Alignment of Gradientscited by 7
- 2024COALA: A Practical and Vision-Centric Federated Learning Platformcited by 7
- 2022Federated Momentum Contrastive Clusteringcited by 6
- 2022Self-supervised On-device Federated Learning from Unlabeled Streamscited by 6
- 2024Robust Training of Federated Models with Extremely Label Deficiencycited by 6
- 2022FedNST: Federated Noisy Student Training for Automatic Speech Recognitioncited by 5
- 2023FSAR: Federated Skeleton-based Action Recognition with Adaptive Topology Structure and Knowledge Distillationcited by 5
- 2024FedSC: Provable Federated Self-supervised Learning with Spectral Contrastive Objective over Non-i.i.d. Datacited by 5
- 2025You Are Your Own Best Teacher: Achieving Centralized-level Performance in Federated Learning under Heterogeneous and Long-tailed Datacited by 5
- 2023A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniquescited by 4
- 2023Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient Harmonizationcited by 4
- 2023Fed-QSSL: A Framework for Personalized Federated Learning under Bitwidth and Data Heterogeneitycited by 4
- 2022Pseudo-Data based Self-Supervised Federated Learning for Classification of Histopathological Imagescited by 3
- 2023FedIL: Federated Incremental Learning from Decentralized Unlabeled Data with Convergence Analysiscited by 3
- 2023FedMAE: Federated Self-Supervised Learning with One-Block Masked Auto-Encodercited by 3
- 2024CCFC: Bridging Federated Clustering and Contrastive Learningcited by 3
- 2025FedDifRC: Unlocking the Potential of Text-to-Image Diffusion Models in Heterogeneous Federated Learningcited by 3
- 2022Federated Training of Dual Encoding Models on Small Non-IID Client Datasetscited by 2
- 2022PATS: Sensitivity-aware Noisy Learning for Pretrained Language Modelscited by 2
- 2023Fast FixMatch: Faster Semi-Supervised Learning with Curriculum Batch Sizecited by 2
- 2024Calibre: Towards Fair and Accurate Personalized Federated Learning with Self-Supervised Learningcited by 2
- 2025Federated Foundation Model for GI Endoscopy Imagescited by 2
- 2022Smart Multi-tenant Federated Learningcited by 1
- 2022Feature Correlation-guided Knowledge Transfer for Federated Self-supervised Learningcited by 1
- 2023Harnessing the Power of Federated Learning in Federated Contextual Banditscited by 1
- 2024Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneitycited by 1
50 most cited of 71 All 71 (JSON)
- 2026Coordination on a Budget: Federated Active Learning with Few Labels19 Aug 2026
- 2026Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Data2 Jul 2026
- 2026Breaking the Prototype Bias Loop: Confidence-Aware Federated Contrastive Learning for Highly Imbalanced Clients3 Mar 2026
- 2025FedHUG: Federated Heterogeneous Unsupervised Generalization for Remote Physiological Measurements14 Oct 2025
- 2025Cooperative Pseudo Labeling for Unsupervised Federated Classification11 Oct 2025
- 2025Federated Self-Supervised Modulation Classification under Non-IID and Imbalanced Data6 Oct 2025
- 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights5 Sep 2025
- 2025IPBA: Imperceptible Perturbation Backdoor Attack in Federated Self-Supervised Learning11 Aug 2025
- 2025Soft Separation and Distillation: Toward Global Uniformity in Federated Unsupervised Learning2 Aug 2025
- 2025FedDifRC: Unlocking the Potential of Text-to-Image Diffusion Models in Heterogeneous Federated Learning9 Jul 2025
- 2025Federated Foundation Model for GI Endoscopy Images30 May 2025
- 2025FSSUAVL: A Discriminative Framework using Vision Models for Federated Self-Supervised Audio and Image Understanding13 Apr 2025
38 more
- 2025Federated Self-Supervised Learning for One-Shot Cross-Modal and Cross-Imaging Technique Segmentation30 Mar 2025
- 2025SAFE: Self-Adjustment Federated Learning Framework for Remote Sensing Collaborative Perception25 Mar 2025
- 2025You Are Your Own Best Teacher: Achieving Centralized-level Performance in Federated Learning under Heterogeneous and Long-tailed Data10 Mar 2025
- 2024Calibre: Towards Fair and Accurate Personalized Federated Learning with Self-Supervised Learning28 Dec 2024
- 2024COALA: A Practical and Vision-Centric Federated Learning Platform23 Jul 2024
- 2024Learning Unlabeled Clients Divergence for Federated Semi-Supervised Learning via Anchor Model Aggregation14 Jul 2024
- 2024A deep cut into Split Federated Self-supervised Learning12 Jun 2024
- 2024Federated Unsupervised Domain Generalization using Global and Local Alignment of Gradients25 May 2024
- 2024EmInspector: Combating Backdoor Attacks in Federated Self-Supervised Learning Through Embedding Inspection21 May 2024
- 2024Generalized Cauchy-Schwarz Divergence and Its Deep Learning Applications7 May 2024
- 2024FedSC: Provable Federated Self-supervised Learning with Spectral Contrastive Objective over Non-i.i.d. Data7 May 2024
- 2024A Mutual Information Perspective on Federated Contrastive Learning3 May 2024
- 2024Unsupervised Federated Optimization at the Edge: D2D-Enabled Learning without Labels15 Apr 2024
- 2024Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity5 Apr 2024
- 2024Rethinking the Representation in Federated Unsupervised Learning with Non-IID Data25 Mar 2024
- 2024FedMef: Towards Memory-efficient Federated Dynamic Pruning21 Mar 2024
- 2024Robust Training of Federated Models with Extremely Label Deficiency22 Feb 2024
- 2024Federated Learning with New Knowledge: Fundamentals, Advances, and Futures3 Feb 2024
- 2024Resource-efficient Layer-wise Federated Self-supervised Learning22 Jan 2024
- 2024CCFC: Bridging Federated Clustering and Contrastive Learning12 Jan 2024
- 2024Relaxed Contrastive Learning for Federated Learning10 Jan 2024
- 2023Harnessing the Power of Federated Learning in Federated Contextual Bandits26 Dec 2023
- 2023Fed-QSSL: A Framework for Personalized Federated Learning under Bitwidth and Data Heterogeneity20 Dec 2023
- 2023MergeSFL: Split Federated Learning with Feature Merging and Batch Size Regulation22 Nov 2023
- 2023Federated Self-Supervised Learning of Monocular Depth Estimators for Autonomous Vehicles7 Oct 2023
- 2023Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient Harmonization13 Sep 2023
- 2023Federated Learning for Large-Scale Scene Modeling with Neural Radiance Fields12 Sep 2023
- 2023Fast FixMatch: Faster Semi-Supervised Learning with Curriculum Batch Size7 Sep 2023
- 2023Federated Learning for Computer Vision24 Aug 2023
- 2023NeFL: Nested Model Scaling for Federated Learning with System Heterogeneous Clients15 Aug 2023
- 2023MAS: Towards Resource-Efficient Federated Multiple-Task Learning21 Jul 2023
- 2023L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learning14 Jul 2023
- 2023Combating Data Imbalances in Federated Semi-supervised Learning with Dual Regulators11 Jul 2023
- 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions27 Jun 2023
- 2023FSAR: Federated Skeleton-based Action Recognition with Adaptive Topology Structure and Knowledge Distillation19 Jun 2023
- 2023FedWon: Triumphing Multi-domain Federated Learning Without Normalization9 Jun 2023
- 2023Personalized Federated Learning under Mixture of Distributions1 May 2023
- 2023Federated Learning without Full Labels: A Survey25 Mar 2023
50 newest of 71 All 71 (JSON)
- 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 83
- 2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
- 2022Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imagingcited by 39
- 2023Personalized Federated Learning under Mixture of Distributionscited by 37
- 2024Relaxed Contrastive Learning for Federated Learningcited by 32
- 2023L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learningcited by 29
- 2023Federated Learning for Computer Visioncited by 22
- 2022A Hybrid Self-Supervised Learning Framework for Vertical Federated Learningcited by 17
- 2022Federated Self-supervised Learning for Heterogeneous Clientscited by 16
- 2022Distributed Contrastive Learning for Medical Image Segmentationcited by 16
- 2024FedMef: Towards Memory-efficient Federated Dynamic Pruningcited by 14
- 2023Re-thinking Federated Active Learning based on Inter-class Diversitycited by 13
12 most cited of 69 All 69 (JSON)
- 2026Coordination on a Budget: Federated Active Learning with Few Labels19 Aug 2026
- 2026Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Data2 Jul 2026
- 2026Breaking the Prototype Bias Loop: Confidence-Aware Federated Contrastive Learning for Highly Imbalanced Clients3 Mar 2026
- 2025FedHUG: Federated Heterogeneous Unsupervised Generalization for Remote Physiological Measurements14 Oct 2025
- 2025Cooperative Pseudo Labeling for Unsupervised Federated Classification11 Oct 2025
- 2025Federated Self-Supervised Modulation Classification under Non-IID and Imbalanced Data6 Oct 2025
- 2025Foundational Models and Federated Learning: Survey, Taxonomy, Challenges and Practical Insights5 Sep 2025
- 2025IPBA: Imperceptible Perturbation Backdoor Attack in Federated Self-Supervised Learning11 Aug 2025
- 2025Soft Separation and Distillation: Toward Global Uniformity in Federated Unsupervised Learning2 Aug 2025
- 2025Federated Foundation Model for GI Endoscopy Images30 May 2025
- 2025FSSUAVL: A Discriminative Framework using Vision Models for Federated Self-Supervised Audio and Image Understanding13 Apr 2025
- 2025Federated Self-Supervised Learning for One-Shot Cross-Modal and Cross-Imaging Technique Segmentation30 Mar 2025
12 newest of 69 All 69 (JSON)