mlsanityData to 14 Oct 2026

A Mutual Information Perspective on Federated Contrastive Learning

arXiv 2405.02081 ↗First posted 3 May 2024cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

We investigate contrastive learning in the federated setting through the lens of SimCLR and multi-view mutual information maximization. In doing so, we uncover a connection between contrastive representation learning and user verification; by adding a user verification loss to each client's local SimCLR loss we recover a lower bound to the global multi-view mutual information. To accommodate for the case of when some labelled data are available at the clients, we extend our SimCLR variant to the federated semi-supervised setting. We see that a supervised SimCLR objective can be obtained with two changes: a) the contrastive loss is computed between datapoints that share the same label and b) we require an additional auxiliary head that predicts the correct labels from either of the two views. Along with the proposed SimCLR extensions, we also study how different sources of non-i.i.d.-ness can impact the performance of federated unsupervised learning through global mutual information maximization; we find that a global objective is beneficial for some sources of non-i.i.d.-ness but can be detrimental for others. We empirically evaluate our proposed extensions in various tasks to validate our claims and furthermore demonstrate that our proposed modifications generalize to other pretraining methods.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 31 of its 34 references that are in the map. First posted on arXiv 3 May 2024.

Builds on

31 of 34 references.

References · 31

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  9. 2018Group Normalizationcited by 1,524
  10. 2020Adaptive Federated Optimizationcited by 1,046
  11. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  12. 2019On Variational Bounds of Mutual Informationcited by 559
19 more
  1. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
  2. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  3. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 264
  4. 2020Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learningcited by 112
  5. 2020Federated Unsupervised Representation Learningcited by 93
  6. 2019Split Learning for collaborative deep learning in healthcarecited by 73
  7. 2022Divergence-aware Federated Self-Supervised Learningcited by 73
  8. 2020Federated Learning with Only Positive Labelscited by 55
  9. 2020On Mutual Information in Contrastive Learning for Visual Representationscited by 54
  10. 2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
  11. 2021Federated Contrastive Learning for Decentralized Unlabeled Medical Imagescited by 34
  12. 2022RSCFed: Random Sampling Consensus Federated Semi-supervised Learningcited by 32
  13. 2021SemiFed: Semi-supervised Federated Learning with Consistency and Pseudo-Labelingcited by 29
  14. 2021Decomposed Mutual Information Estimation for Contrastive Representation Learningcited by 25
  15. 2022Federated Learning from Only Unlabeled Data with Class-Conditional-Sharing Clientscited by 20
  16. 2022Federated Self-supervised Learning for Heterogeneous Clientscited by 16
  17. 2022Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?cited by 12
  18. 2021Federated Learning of User Verification Models Without Sharing Embeddingscited by 11
  19. 2022Federated Semi-Supervised Learning with Prototypical Networkscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025Federated Learning with Sample-level Client Drift Mitigationcited by 6
  2. 2026Toward Enhancing Representation Learning in Federated Multi-Task Settingscited by 2
  3. 2025How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reductioncited by 1
  4. 2025FedQuad: Federated Stochastic Quadruplet Learning to Mitigate Data Heterogeneitycited by 1
  5. 2025Maximally Useful and Minimally Redundant: The Key to Self Supervised Learning for Imbalanced Datacited by 1
  6. 2026Enhancing Federated Quadruplet Learning: Stochastic Client Selection and Embedding Stability Analysiscited by 0
  7. 2026Coordination on a Budget: Federated Active Learning with Few Labelscited by 0