A deep cut into Split Federated Self-supervised Learning
Not on the map. Why?
Abstract (arXiv)
Collaborative self-supervised learning has recently become feasible in highly distributed environments by dividing the network layers between client devices and a central server. However, state-of-the-art methods, such as MocoSFL, are optimized for network division at the initial layers, which decreases the protection of the client data and increases communication overhead. In this paper, we demonstrate that splitting depth is crucial for maintaining privacy and communication efficiency in distributed training. We also show that MocoSFL suffers from a catastrophic quality deterioration for the minimal communication overhead. As a remedy, we introduce Momentum-Aligned contrastive Split Federated Learning (MonAcoSFL), which aligns online and momentum client models during training procedure. Consequently, we achieve state-of-the-art accuracy while significantly reducing the communication overhead, making MonAcoSFL more practical in real-world scenarios.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 37 references that are in the map. First posted on arXiv 12 Jun 2024.
Builds on
30 of 37 references.
References · 30
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
- 2020Exploring Simple Siamese Representation Learningcited by 2,853
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
18 more
- 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
- 2018Federated Learning with Non-IID Datacited by 1,099
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
- 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
- 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
- 2020SplitFed: When Federated Learning Meets Split Learningcited by 308
- 2021FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Spacecited by 284
- 2023A Cookbook of Self-Supervised Learningcited by 248
- 2021FedGNN: Federated Graph Neural Network for Privacy-Preserving Recommendationcited by 97
- 2020Federated Unsupervised Representation Learningcited by 92
- 2022Divergence-aware Federated Self-Supervised Learningcited by 71
- 2021Collaborative Unsupervised Visual Representation Learning from Decentralized Datacited by 68
- 2020Vertically Federated Graph Neural Network for Privacy-Preserving Node Classificationcited by 61
- 2021Improving Transferability of Representations via Augmentation-Aware Self-Supervisioncited by 48
- 2022Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learningcited by 25
- 2022Federated Self-supervised Learning for Heterogeneous Clientscited by 16
- 2023Exploring the Privacy-Energy Consumption Tradeoff for Split Federated Learningcited by 9
- 2021Improving memory banks for unsupervised learning with large mini-batch, consistency and hard negative miningcited by 4
Built on by
Not cited yet.