mlsanityData to 7 Oct 2026

FedSC: Provable Federated Self-supervised Learning with Spectral Contrastive Objective over Non-i.i.d. Data

arXiv 2405.03949 ↗First posted 7 May 2024cs.LG cs.CR eess.SPcited by 5

Not on the map. Why?

Abstract (arXiv)

Recent efforts have been made to integrate self-supervised learning (SSL) with the framework of federated learning (FL). One unique challenge of federated self-supervised learning (FedSSL) is that the global objective of FedSSL usually does not equal the weighted sum of local SSL objectives. Consequently, conventional approaches, such as federated averaging (FedAvg), fail to precisely minimize the FedSSL global objective, often resulting in suboptimal performance, especially when data is non-i.i.d.. To fill this gap, we propose a provable FedSSL algorithm, named FedSC, based on the spectral contrastive objective. In FedSC, clients share correlation matrices of data representations in addition to model weights periodically, which enables inter-client contrast of data samples in addition to intra-client contrast and contraction, resulting in improved quality of data representations. Differential privacy (DP) protection is deployed to control the additional privacy leakage on local datasets when correlation matrices are shared. We also provide theoretical analysis on the convergence and extra privacy leakage. The experimental results validate the effectiveness of our proposed algorithm.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 25 of its 31 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

25 of 31 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  8. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  9. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  10. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  11. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  12. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 528
13 more
  1. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
  2. 2019On Mutual Information Maximization for Representation Learningcited by 315
  3. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  4. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  5. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  6. 2020LDP-Fed: Federated Learning with Local Differential Privacycited by 129
  7. 2020Federated Unsupervised Representation Learningcited by 92
  8. 2021Differentially Private Federated Learning on Heterogeneous Datacited by 74
  9. 2022Divergence-aware Federated Self-Supervised Learningcited by 71
  10. 2021Collaborative Unsupervised Visual Representation Learning from Decentralized Datacited by 68
  11. 2022How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learningcited by 65
  12. 2022FedX: Unsupervised Federated Learning with Cross Knowledge Distillationcited by 40
  13. 2022Implicit variance regularization in non-contrastive SSLcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Towards Efficient Federated Learning of Networked Mixture-of-Experts for Mobile Edge Computingcited by 1
  2. 2026Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Datacited by 1
  3. 2025HFedCKD: Toward Robust Heterogeneous Federated Learning via Data-free Knowledge Distillation and Two-way Contrastcited by 0
  4. 2026Breaking the Prototype Bias Loop: Confidence-Aware Federated Contrastive Learning for Highly Imbalanced Clientscited by 0
  5. 2026Coordination on a Budget: Federated Active Learning with Few Labelscited by 0