mlsanityData to 7 Oct 2026

Federated Learning of a Mixture of Global and Local Models

arXiv 2002.05516 ↗First posted 10 Feb 2020cs.LG cs.DC math.OCcited by 208

Not on the map. Why?

Abstract (arXiv)

We propose a new optimization formulation for training federated learning models. The standard formulation has the form of an empirical risk minimization problem constructed to find a single global model trained from the private data stored across all participating devices. In contrast, our formulation seeks an explicit trade-off between this traditional global model and the local models, which can be learned by each device from its own private data without any communication. Further, we develop several efficient variants of SGD (with and without partial participation and with and without variance reduction) for solving the new formulation and prove communication complexity guarantees. Notably, our methods are similar but not identical to federated averaging / local SGD, thus shedding some light on the role of local steps in federated learning. In particular, we are the first to i) show that local steps can improve communication for problems with heterogeneous data, and ii) point out that personalization yields reduced communication complexity.

Key facts. Cited by 208 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 47; 2022: 50; 2023: 42; 2024: 34; 2025: 24; 2026: 9.

208 papers.

Sort:

By area:

  1. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  2. 2021Towards Personalized Federated Learningcited by 462
  3. 2021Federated Learning on Non-IID Data: A Surveycited by 345
  4. 2021A Field Guide to Federated Optimizationcited by 312
  5. 2021Personalized Federated Learning using Hypernetworkscited by 210
  6. 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 189
  7. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 180
  8. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
  9. 2021Federated Learning for Open Bankingcited by 138
  10. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  11. 2021Multi-Center Federated Learning: Clients Clustering for Better Personalizationcited by 112
  12. 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 108
38 more
  1. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  2. 2021Federated Reconstruction: Partially Local Federated Learningcited by 78
  3. 2021Personalized Federated Learning with Gaussian Processescited by 76
  4. 2022Personalized Federated Learning via Variational Bayesian Inferencecited by 68
  5. 2023Advances and Challenges in Meta-Learning: A Technical Reviewcited by 65
  6. 2021Federated Multi-armed Bandits with Personalizationcited by 62
  7. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  8. 2021Personalized Federated Learning through Local Memorizationcited by 57
  9. 2021Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learningcited by 53
  10. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  11. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  12. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 45
  13. 2022pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learningcited by 44
  14. 2022A Survey on Heterogeneous Federated Learningcited by 42
  15. 2023Recent Advances on Federated Learning: A Systematic Surveycited by 40
  16. 2022Test-Time Robust Personalization for Federated Learningcited by 39
  17. 2023Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearningcited by 39
  18. 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
  19. 2021Non-IID data and Continual Learning processes in Federated Learning: A long road aheadcited by 36
  20. 2022On Privacy and Personalization in Cross-Silo Federated Learningcited by 36
  21. 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
  22. 2022FedPop: A Bayesian Approach for Personalised Federated Learningcited by 35
  23. 2021Personalized Federated Learning: A Unified Framework and Universal Optimization Techniquescited by 34
  24. 2022CD2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learningcited by 34
  25. 2024Towards Efficient Replay in Federated Incremental Learningcited by 33
  26. 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learningcited by 32
  27. 2022Motley: Benchmarking Heterogeneity and Personalization in Federated Learningcited by 31
  28. 2022Achieving Personalized Federated Learning with Sparse Local Modelscited by 30
  29. 2021Federated f-Differential Privacycited by 29
  30. 2022Disentangled Federated Learning for Tackling Attributes Skew via Invariant Aggregation and Diversity Transferringcited by 28
  31. 2021A New Look and Convergence Rate of Federated Multi-Task Learning with Laplacian Regularizationcited by 27
  32. 2021Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneitycited by 27
  33. 2023Privacy and Fairness in Federated Learning: on the Perspective of Trade-offcited by 26
  34. 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillationcited by 25
  35. 2020Specialized federated learning using a mixture of expertscited by 23
  36. 2021Personalized Federated Learning with Contextualized Generalizationcited by 23
  37. 2021Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learningcited by 23
  38. 2021The Internet of Federated Things (IoFT): A Vision for the Future and In-depth Survey of Data-driven Approaches for Federated Learningcited by 23

50 most cited of 208 All 208 (JSON)