# A Mutual Information Perspective on Federated Contrastive Learning

arXiv: https://arxiv.org/abs/2405.02081 · HTML: https://mlsanity.com/paper/2405.02081/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 34 references that are in the map.
- First posted on arXiv 3 May 2024.

## Builds on (most cited first, 30 of 31)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss](https://mlsanity.com/paper/2106.04156/) (2021)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learning](https://mlsanity.com/paper/2006.12097/) (2020)
- [Federated Unsupervised Representation Learning](https://mlsanity.com/paper/2010.08982/) (2020)
- [Split Learning for collaborative deep learning in healthcare](https://mlsanity.com/paper/1912.12115/) (2019)
- [Divergence-aware Federated Self-Supervised Learning](https://mlsanity.com/paper/2204.04385/) (2022)
- [Federated Learning with Only Positive Labels](https://mlsanity.com/paper/2004.10342/) (2020)
- [On Mutual Information in Contrastive Learning for Visual Representations](https://mlsanity.com/paper/2005.13149/) (2020)
- [Orchestra: Unsupervised Federated Learning via Globally Consistent Clustering](https://mlsanity.com/paper/2205.11506/) (2022)
- [Federated Contrastive Learning for Decentralized Unlabeled Medical Images](https://mlsanity.com/paper/2109.07504/) (2021)
- [RSCFed: Random Sampling Consensus Federated Semi-supervised Learning](https://mlsanity.com/paper/2203.13993/) (2022)
- [SemiFed: Semi-supervised Federated Learning with Consistency and Pseudo-Labeling](https://mlsanity.com/paper/2108.09412/) (2021)
- [Decomposed Mutual Information Estimation for Contrastive Representation Learning](https://mlsanity.com/paper/2106.13401/) (2021)
- [Federated Learning from Only Unlabeled Data with Class-Conditional-Sharing Clients](https://mlsanity.com/paper/2204.03304/) (2022)
- [Federated Self-supervised Learning for Heterogeneous Clients](https://mlsanity.com/paper/2205.12493/) (2022)
- [Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?](https://mlsanity.com/paper/2210.10947/) (2022)
- [Federated Learning of User Verification Models Without Sharing Embeddings](https://mlsanity.com/paper/2104.08776/) (2021)

## Built on by (most cited first, 7 of 7)

- [Federated Learning with Sample-level Client Drift Mitigation](https://mlsanity.com/paper/2501.11360/) (2025)
- [Toward Enhancing Representation Learning in Federated Multi-Task Settings](https://mlsanity.com/paper/2602.01626/) (2026)
- [How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction](https://mlsanity.com/paper/2507.11161/) (2025)
- [FedQuad: Federated Stochastic Quadruplet Learning to Mitigate Data Heterogeneity](https://mlsanity.com/paper/2509.04107/) (2025)
- [Maximally Useful and Minimally Redundant: The Key to Self Supervised Learning for Imbalanced Data](https://mlsanity.com/paper/2509.08469/) (2025)
- [Enhancing Federated Quadruplet Learning: Stochastic Client Selection and Embedding Stability Analysis](https://mlsanity.com/paper/2605.07888/) (2026)
- [Coordination on a Budget: Federated Active Learning with Few Labels](https://mlsanity.com/paper/2608.18634/) (2026)

## Abstract (arXiv)

We investigate contrastive learning in the federated setting through the lens of SimCLR and multi-view mutual information maximization. In doing so, we uncover a connection between contrastive representation learning and user verification; by adding a user verification loss to each client's local SimCLR loss we recover a lower bound to the global multi-view mutual information. To accommodate for the case of when some labelled data are available at the clients, we extend our SimCLR variant to the federated semi-supervised setting. We see that a supervised SimCLR objective can be obtained with two changes: a) the contrastive loss is computed between datapoints that share the same label and b) we require an additional auxiliary head that predicts the correct labels from either of the two views. Along with the proposed SimCLR extensions, we also study how different sources of non-i.i.d.-ness can impact the performance of federated unsupervised learning through global mutual information maximization; we find that a global objective is beneficial for some sources of non-i.i.d.-ness but can be detrimental for others. We empirically evaluate our proposed extensions in various tasks to validate our claims and furthermore demonstrate that our proposed modifications generalize to other pretraining methods.