mlsanityData to 7 Oct 2026

Model-Contrastive Federated Learning

arXiv 2103.16257 ↗First posted 30 Mar 2021cs.LG cs.AI cs.CVcited by 762

Not on the map. Why?

Abstract (arXiv)

Federated learning enables multiple parties to collaboratively train a machine learning model without communicating their local data. A key challenge in federated learning is to handle the heterogeneity of local data distribution across parties. Although many studies have been proposed to address this challenge, we find that they fail to achieve high performance in image datasets with deep learning models. In this paper, we propose MOON: model-contrastive federated learning. MOON is a simple and effective federated learning framework. The key idea of MOON is to utilize the similarity between model representations to correct the local training of individual parties, i.e., conducting contrastive learning in model-level. Our extensive experiments show that MOON significantly outperforms the other state-of-the-art federated learning algorithms on various image classification tasks.

Key facts. Cited by 762 papers in the map (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 30 Mar 2021.

Builds on

37 of 49 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2019Advances and Open Problems in Federated Learningcited by 3,021
  10. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  11. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  12. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
25 more
  1. 2019Contrastive Multiview Codingcited by 1,297
  2. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  3. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  4. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  5. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  6. 2018LEAF: A Benchmark for Federated Settingscited by 780
  7. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  8. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  9. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  10. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  11. 2020Federated Learning with Matched Averagingcited by 564
  12. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  13. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 371
  14. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 298
  15. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 282
  16. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  17. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  18. 2020Federated Visual Classification with Real-World Data Distributioncited by 126
  19. 2020Lower Bounds and Optimal Algorithms for Personalized Federated Learningcited by 122
  20. 2020FedVision: An Online Visual Object Detection Platform Powered by Federated Learningcited by 121
  21. 2020Federated Unsupervised Representation Learningcited by 92
  22. 2020Federated Bayesian Optimization via Thompson Samplingcited by 77
  23. 2020Blockchain-Federated-Learning and Deep Learning Models for COVID-19 detection using CT Imagingcited by 56
  24. 2019Practical Federated Gradient Boosting Decision Treescited by 53
  25. 2020The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systemscited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 91; 2023: 160; 2024: 193; 2025: 179; 2026: 114.

762 papers.

Sort:

By area:

  1. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  2. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  3. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  4. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  5. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
  6. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 141
  7. 2022FedALA: Adaptive Local Aggregation for Personalized Federated Learningcited by 121
  8. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  9. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  10. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  11. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  12. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
38 more
  1. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  2. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
  3. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 85
  4. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  5. 2021FedProc: Prototypical Contrastive Federated Learning on Non-IID datacited by 80
  6. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  7. 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 66
  8. 2023Advances and Challenges in Meta-Learning: A Technical Reviewcited by 65
  9. 2021HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Imagescited by 62
  10. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 55
  11. 2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 54
  12. 2022Personalized Subgraph Federated Learningcited by 54
  13. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  14. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
  15. 2022Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learningcited by 52
  16. 2022FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Datacited by 49
  17. 2022Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learningcited by 45
  18. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 45
  19. 2024FedGTA: Topology-aware Averaging for Federated Graph Learningcited by 44
  20. 2021Federated Learning with Domain Generalizationcited by 41
  21. 2022FedX: Unsupervised Federated Learning with Cross Knowledge Distillationcited by 40
  22. 2023Learning Federated Visual Prompt in Null Space for MRI Reconstructioncited by 40
  23. 2021FedCG: Leverage Conditional GAN for Protecting Privacy and Maintaining Competitive Performance in Federated Learningcited by 39
  24. 2022Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imagingcited by 39
  25. 2023Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearningcited by 39
  26. 2023Generalizable Heterogeneous Federated Cross-Correlation and Instance Similarity Learningcited by 39
  27. 2023Eliminating Domain Bias for Federated Learning in Representation Spacecited by 38
  28. 2023FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policycited by 36
  29. 2021Specificity-Preserving Federated Learning for MR Image Reconstructioncited by 35
  30. 2022FedBR: Improving Federated Learning on Heterogeneous Data via Local Learning Bias Reductioncited by 35
  31. 2023Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Datacited by 35
  32. 2022CD2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learningcited by 34
  33. 2022Multi-Level Branched Regularization for Federated Learningcited by 34
  34. 2022Federated Learning for Medical Applications: A Taxonomy, Current Trends, Challenges, and Future Research Directionscited by 34
  35. 2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformerscited by 33
  36. 2023FedNoRo: Towards Noise-Robust Federated Learning by Addressing Class Imbalance and Label Noise Heterogeneitycited by 33
  37. 2024An Aggregation-Free Federated Learning for Tackling Data Heterogeneitycited by 33
  38. 2021Differentially Private Federated Bayesian Optimization with Distributed Explorationcited by 32

50 most cited of 762 All 762 (JSON)