mlsanityData to 7 Oct 2026

A Field Guide to Federated Optimization

arXiv 2107.06917 ↗First posted 14 Jul 2021cs.LGcited by 312

Not on the map. Why?

Abstract (arXiv)

Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection. The distributed learning process can be formulated as solving federated optimization problems, which emphasize communication efficiency, data heterogeneity, compatibility with privacy and system requirements, and other constraints that are not primary considerations in other problem settings. This paper provides recommendations and guidelines on formulating, designing, evaluating and analyzing federated optimization algorithms through concrete examples and practical implementation, with a focus on conducting effective simulations to infer real-world performance. The goal of this work is not to survey the current literature, but to inspire researchers and practitioners to design federated learning algorithms that can be used in various practical applications.

Key facts. Cited by 312 papers in the map (data through 7 Oct 2026). Builds on 195 of its 260 references that are in the map. First posted on arXiv 14 Jul 2021.

Builds on

195 of 260 references.

References · 195

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  8. 2016Deep Learning with Differential Privacycited by 2,682
  9. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  10. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  11. 2018Group Normalizationcited by 1,514
  12. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
38 more
  1. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  2. 2020Adaptive Federated Optimizationcited by 1,036
  3. 2019Deep Leakage from Gradientscited by 1,004
  4. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  6. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  7. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  8. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  9. 2019Towards Federated Learning at Scale: System Designcited by 826
  10. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  11. 2018LEAF: A Benchmark for Federated Settingscited by 780
  12. 2018How To Backdoor Federated Learningcited by 745
  13. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 698
  14. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  15. 2017Federated Multi-Task Learningcited by 669
  16. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  17. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  18. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  19. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  20. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  21. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  22. 2019Federated Learning with Personalization Layerscited by 533
  23. 2020An Efficient Framework for Clustered Federated Learningcited by 505
  24. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
  25. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  26. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 479
  27. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  28. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  29. 2019Agnostic Federated Learningcited by 451
  30. 2019Federated Machine Learning: Concept and Applicationscited by 409
  31. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 407
  32. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 391
  33. 2019Fair Resource Allocation in Federated Learningcited by 389
  34. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  35. 2018Sparsified SGD with Memorycited by 373
  36. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 363
  37. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  38. 2020Adaptive Personalized Federated Learningcited by 353

50 most cited of 195 All 195 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 38; 2022: 88; 2023: 89; 2024: 54; 2025: 30; 2026: 13.

312 papers.

Sort:

By area:

  1. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  2. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  3. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  4. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  5. 2021FairFed: Enabling Group Fairness in Federated Learningcited by 114
  6. 2022Federated Learning with Partial Model Personalizationcited by 114
  7. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 114
  8. 2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 108
  9. 2021Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Modelscited by 106
  10. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  11. 2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 96
  12. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
38 more
  1. 2021Differentially Private Federated Learning on Heterogeneous Datacited by 74
  2. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  3. 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 66
  4. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 64
  5. 2021LightSecAgg: a Lightweight and Versatile Design for Secure Aggregation in Federated Learningcited by 63
  6. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  7. 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 62
  8. 2022Client Selection in Federated Learning: Principles, Challenges, and Opportunitiescited by 62
  9. 2022Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learningcited by 59
  10. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 57
  11. 2021EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learningcited by 53
  12. 2021Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learningcited by 53
  13. 2022FederatedScope: A Flexible Federated Learning Platform for Heterogeneitycited by 53
  14. 2022Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Modelscited by 52
  15. 2022Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learningcited by 52
  16. 2022FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Datacited by 49
  17. 2023Improving the Model Consistency of Decentralized Federated Learningcited by 45
  18. 2023Fusing Models with Complementary Expertisecited by 45
  19. 2021Anarchic Federated Learningcited by 42
  20. 2022FedX: Unsupervised Federated Learning with Cross Knowledge Distillationcited by 40
  21. 2023Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challengescited by 39
  22. 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
  23. 2022Private Adaptive Optimization with Side Informationcited by 38
  24. 2022Gradient Obfuscation Gives a False Sense of Security in Federated Learningcited by 37
  25. 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Datacited by 36
  26. 2022FedPop: A Bayesian Approach for Personalised Federated Learningcited by 35
  27. 2022SoteriaFL: A Unified Framework for Private Federated Learning with Communication Compressioncited by 34
  28. 2022Multi-Level Branched Regularization for Federated Learningcited by 34
  29. 2022Federated Learning of Generative Image Priors for MRI Reconstructioncited by 33
  30. 2022Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learningcited by 33
  31. 2024An Aggregation-Free Federated Learning for Tackling Data Heterogeneitycited by 33
  32. 2021FedCV: A Federated Learning Framework for Diverse Computer Vision Taskscited by 32
  33. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 31
  34. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  35. 2022On the Convergence of Clustered Federated Learningcited by 31
  36. 2022Motley: Benchmarking Heterogeneity and Personalization in Federated Learningcited by 31
  37. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  38. 2023On the Convergence of Federated Averaging with Cyclic Client Participationcited by 31

50 most cited of 312 All 312 (JSON)