# Relaxed Contrastive Learning for Federated Learning

arXiv: https://arxiv.org/abs/2401.04928 · HTML: https://mlsanity.com/paper/2401.04928/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 50 references that are in the map.
- First posted on arXiv 10 Jan 2024.

## Builds on (most cited first, 30 of 41)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization](https://mlsanity.com/paper/1909.13014/) (2019)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [SCAFFOLD: Stochastic Controlled Averaging for Federated Learning](https://mlsanity.com/paper/1910.06378/) (2019)
- [Federated Learning with Compression: Unified Analysis and Sharp Guarantees](https://mlsanity.com/paper/2007.01154/) (2020)
- [FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2108.06098/) (2021)
- [Federated Visual Classification with Real-World Data Distribution](https://mlsanity.com/paper/2003.08082/) (2020)
- [Federated Learning with Label Distribution Skew via Logits Calibration](https://mlsanity.com/paper/2209.00189/) (2022)
- [FedMix: Approximation of Mixup under Mean Augmented Federated Learning](https://mlsanity.com/paper/2107.00233/) (2021)
- [Preservation of the Global Knowledge by Not-True Distillation in Federated Learning](https://mlsanity.com/paper/2106.03097/) (2021)
- [FedProc: Prototypical Contrastive Federated Learning on Non-IID data](https://mlsanity.com/paper/2109.12273/) (2021)
- [Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms](https://mlsanity.com/paper/2010.05273/) (2020)
- [FedCM: Federated Learning with Client-level Momentum](https://mlsanity.com/paper/2106.10874/) (2021)

## Built on by (most cited first, 30 of 32)

- [Robust Asymmetric Heterogeneous Federated Learning with Corrupted Clients](https://mlsanity.com/paper/2503.09206/) (2025)
- [CCFC: Bridging Federated Clustering and Contrastive Learning](https://mlsanity.com/paper/2401.06634/) (2024)
- [FedCoSR: Personalized Federated Learning with Contrastive Shareable Representations for Label Heterogeneity in Non-IID Data](https://mlsanity.com/paper/2404.17916/) (2024)
- [FedSC: Federated Learning with Semantic-Aware Collaboration](https://mlsanity.com/paper/2506.21012/) (2025)
- [FedDifRC: Unlocking the Potential of Text-to-Image Diffusion Models in Heterogeneous Federated Learning](https://mlsanity.com/paper/2507.06482/) (2025)
- [Toward Enhancing Representation Learning in Federated Multi-Task Settings](https://mlsanity.com/paper/2602.01626/) (2026)
- [Diffusion-Guided Semantic Consistency for Multimodal Heterogeneity](https://mlsanity.com/paper/2603.19337/) (2026)
- [Collaborative Adaptive Curriculum for Progressive Knowledge Distillation](https://mlsanity.com/paper/2603.20296/) (2026)
- [Federated Cross-Training Learners for Robust Generalization under Data Heterogeneity](https://mlsanity.com/paper/2405.20046/) (2024)
- [FedSKC: Federated Learning with Non-IID Data via Structural Knowledge Collaboration](https://mlsanity.com/paper/2505.18981/) (2025)
- [FedWSQ: Efficient Federated Learning with Weight Standardization and Distribution-Aware Non-Uniform Quantization](https://mlsanity.com/paper/2506.23516/) (2025)
- [FedQuad: Federated Stochastic Quadruplet Learning to Mitigate Data Heterogeneity](https://mlsanity.com/paper/2509.04107/) (2025)
- [Fisher-Informed Parameterwise Aggregation for Federated Learning with Heterogeneous Data](https://mlsanity.com/paper/2601.13608/) (2026)
- [First-Order Softmax Weighted Switching Gradient Method for Distributed Stochastic Minimax Optimization with Stochastic Constraints](https://mlsanity.com/paper/2603.05774/) (2026)
- [ForgeVLA: Federated Vision-Language-Action Learning without Language Annotations](https://mlsanity.com/paper/2605.07474/) (2026)
- [Neighborhood and Global Perturbations Supported SAM in Federated Learning: From Local Tweaks To Global Awareness](https://mlsanity.com/paper/2408.14144/) (2024)
- [Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Data](https://mlsanity.com/paper/2505.20485/) (2025)
- [Decoupled Contrastive Learning for Federated Learning](https://mlsanity.com/paper/2508.04005/) (2025)
- [DART: A Server-side Plug-in for Resource-efficient Robust Federated Learning](https://mlsanity.com/paper/2508.17381/) (2025)
- [Contrastive Self-Supervised Learning at the Edge: An Energy Perspective](https://mlsanity.com/paper/2510.08374/) (2025)
- [Local K-Similarity Constraint for Federated Learning with Label Noise](https://mlsanity.com/paper/2511.06169/) (2025)
- [Federated Vision Transformer with Adaptive Focal Loss for Medical Image Classification](https://mlsanity.com/paper/2602.01633/) (2026)
- [Towards Performance-Enhanced Model-Contrastive Federated Learning using Historical Information in Heterogeneous Scenarios](https://mlsanity.com/paper/2602.11945/) (2026)
- [Breaking the Prototype Bias Loop: Confidence-Aware Federated Contrastive Learning for Highly Imbalanced Clients](https://mlsanity.com/paper/2603.03007/) (2026)
- [Geometric Knowledge-Assisted Federated Dual Knowledge Distillation Approach Towards Remote Sensing Satellite Imagery](https://mlsanity.com/paper/2603.07774/) (2026)
- [Domain-Skewed Federated Learning with Feature Decoupling and Calibration](https://mlsanity.com/paper/2603.14238/) (2026)
- [From Selection to Scheduling: Federated Geometry-Aware Correction Makes Exemplar Replay Work Better under Continual Dynamic Heterogeneity](https://mlsanity.com/paper/2604.08617/) (2026)
- [Enhancing Federated Quadruplet Learning: Stochastic Client Selection and Embedding Stability Analysis](https://mlsanity.com/paper/2605.07888/) (2026)
- [FedHPro: Federated Hyper-Prototype Learning via Gradient Matching](https://mlsanity.com/paper/2605.13475/) (2026)
- [Defending Against Backdoor Attacks via Alignment Checking in Model-Contrastive Federated Learning](https://mlsanity.com/paper/2607.26933/) (2026)

## Abstract (arXiv)

We propose a novel contrastive learning framework to effectively address the challenges of data heterogeneity in federated learning. We first analyze the inconsistency of gradient updates across clients during local training and establish its dependence on the distribution of feature representations, leading to the derivation of the supervised contrastive learning (SCL) objective to mitigate local deviations. In addition, we show that a na"ive adoption of SCL in federated learning leads to representation collapse, resulting in slow convergence and limited performance gains. To address this issue, we introduce a relaxed contrastive learning loss that imposes a divergence penalty on excessively similar sample pairs within each class. This strategy prevents collapsed representations and enhances feature transferability, facilitating collaborative training and leading to significant performance improvements. Our framework outperforms all existing federated learning approaches by huge margins on the standard benchmarks through extensive experimental results.