mlsanityData to 14 Oct 2026

Relaxed Contrastive Learning for Federated Learning

arXiv 2401.04928 ↗First posted 10 Jan 2024cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

We propose a novel contrastive learning framework to effectively address the challenges of data heterogeneity in federated learning. We first analyze the inconsistency of gradient updates across clients during local training and establish its dependence on the distribution of feature representations, leading to the derivation of the supervised contrastive learning (SCL) objective to mitigate local deviations. In addition, we show that a na"ive adoption of SCL in federated learning leads to representation collapse, resulting in slow convergence and limited performance gains. To address this issue, we introduce a relaxed contrastive learning loss that imposes a divergence penalty on excessively similar sample pairs within each class. This strategy prevents collapsed representations and enhances feature transferability, facilitating collaborative training and leading to significant performance improvements. Our framework outperforms all existing federated learning approaches by huge margins on the standard benchmarks through extensive experimental results.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 10 Jan 2024.

Builds on

41 of 50 references.

References · 41

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  9. 2020Supervised Contrastive Learningcited by 2,829
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  11. 2018Group Normalizationcited by 1,524
  12. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
29 more
  1. 2018Federated Learning with Non-IID Datacited by 1,108
  2. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  3. 2020Adaptive Federated Optimizationcited by 1,046
  4. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  5. 2021Model-Contrastive Federated Learningcited by 766
  6. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  7. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 356
  8. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  9. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 175
  10. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 160
  11. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  12. 2020Federated Visual Classification with Real-World Data Distributioncited by 128
  13. 2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 109
  14. 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
  15. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  16. 2021FedProc: Prototypical Contrastive Federated Learning on Non-IID datacited by 81
  17. 2020Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithmscited by 78
  18. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  19. 2020FedDANE: A Federated Newton-Type Methodcited by 76
  20. 2022Divergence-aware Federated Self-Supervised Learningcited by 73
  21. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  22. 2022Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learningcited by 46
  23. 2022FedBR: Improving Federated Learning on Heterogeneous Data via Local Learning Bias Reductioncited by 35
  24. 2022Multi-Level Branched Regularization for Federated Learningcited by 34
  25. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Datacited by 31
  26. 2022Acceleration of Federated Learning with Alleviated Forgetting in Local Trainingcited by 29
  27. 2022Communication-Efficient Federated Learning with Accelerated Client Gradientcited by 24
  28. 2022Discriminability-Transferability Trade-Off: An Information-Theoretic Perspectivecited by 13
  29. 2023Quantifying the Variability Collapse of Neural Networkscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 11; 2026: 17.

32 papers.

Sort:

By area:

  1. 2025Robust Asymmetric Heterogeneous Federated Learning with Corrupted Clientscited by 5
  2. 2024CCFC: Bridging Federated Clustering and Contrastive Learningcited by 4
  3. 2024FedCoSR: Personalized Federated Learning with Contrastive Shareable Representations for Label Heterogeneity in Non-IID Datacited by 3
  4. 2025FedSC: Federated Learning with Semantic-Aware Collaborationcited by 3
  5. 2025FedDifRC: Unlocking the Potential of Text-to-Image Diffusion Models in Heterogeneous Federated Learningcited by 3
  6. 2026Toward Enhancing Representation Learning in Federated Multi-Task Settingscited by 2
  7. 2026Diffusion-Guided Semantic Consistency for Multimodal Heterogeneitycited by 2
  8. 2026Collaborative Adaptive Curriculum for Progressive Knowledge Distillationcited by 2
  9. 2024Federated Cross-Training Learners for Robust Generalization under Data Heterogeneitycited by 1
  10. 2025FedSKC: Federated Learning with Non-IID Data via Structural Knowledge Collaborationcited by 1
  11. 2025FedWSQ: Efficient Federated Learning with Weight Standardization and Distribution-Aware Non-Uniform Quantizationcited by 1
  12. 2025FedQuad: Federated Stochastic Quadruplet Learning to Mitigate Data Heterogeneitycited by 1
20 more
  1. 2026Fisher-Informed Parameterwise Aggregation for Federated Learning with Heterogeneous Datacited by 1
  2. 2026First-Order Softmax Weighted Switching Gradient Method for Distributed Stochastic Minimax Optimization with Stochastic Constraintscited by 1
  3. 2026ForgeVLA: Federated Vision-Language-Action Learning without Language Annotationscited by 1
  4. 2024Neighborhood and Global Perturbations Supported SAM in Federated Learning: From Local Tweaks To Global Awarenesscited by 0
  5. 2025Avoid Forgetting by Preserving Global Knowledge Gradients in Federated Learning with Non-IID Datacited by 0
  6. 2025Decoupled Contrastive Learning for Federated Learningcited by 0
  7. 2025DART: A Server-side Plug-in for Resource-efficient Robust Federated Learningcited by 0
  8. 2025Contrastive Self-Supervised Learning at the Edge: An Energy Perspectivecited by 0
  9. 2025Local K-Similarity Constraint for Federated Learning with Label Noisecited by 0
  10. 2026Federated Vision Transformer with Adaptive Focal Loss for Medical Image Classificationcited by 0
  11. 2026Towards Performance-Enhanced Model-Contrastive Federated Learning using Historical Information in Heterogeneous Scenarioscited by 0
  12. 2026Breaking the Prototype Bias Loop: Confidence-Aware Federated Contrastive Learning for Highly Imbalanced Clientscited by 0
  13. 2026Geometric Knowledge-Assisted Federated Dual Knowledge Distillation Approach Towards Remote Sensing Satellite Imagerycited by 0
  14. 2026Domain-Skewed Federated Learning with Feature Decoupling and Calibrationcited by 0
  15. 2026From Selection to Scheduling: Federated Geometry-Aware Correction Makes Exemplar Replay Work Better under Continual Dynamic Heterogeneitycited by 0
  16. 2026Enhancing Federated Quadruplet Learning: Stochastic Client Selection and Embedding Stability Analysiscited by 0
  17. 2026FedHPro: Federated Hyper-Prototype Learning via Gradient Matchingcited by 0
  18. 2026Defending Against Backdoor Attacks via Alignment Checking in Model-Contrastive Federated Learningcited by 0
  19. 2026Coordination on a Budget: Federated Active Learning with Few Labelscited by 0
  20. 2026PACE: Propagation-Aware Collaborative Correction for One-Shot Personalized Federated Graph Learningcited by 0