mlsanityData to 7 Oct 2026

Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learning

arXiv 2103.05032 ↗First posted 8 Mar 2021cs.LG cs.DC math.OCcited by 52

Not on the map. Why?

Abstract (arXiv)

We study a family of algorithms, which we refer to as local update methods, generalizing many federated and meta-learning algorithms. We prove that for quadratic models, local update methods are equivalent to first-order optimization on a surrogate loss we exactly characterize. Moreover, fundamental algorithmic choices (such as learning rates) explicitly govern a trade-off between the condition number of the surrogate loss and its alignment with the true loss. We derive novel convergence rates showcasing these trade-offs and highlight their importance in communication-limited settings. Using these insights, we are able to compare local update methods based on their convergence/accuracy trade-off, not just their convergence to critical points of the empirical loss. Our results shed new light on a broad range of phenomena, including the efficacy of server momentum in federated learning and the impact of proximal client updates.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 8 Mar 2021.

Builds on

32 of 41 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2019Advances and Open Problems in Federated Learningcited by 3,021
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  6. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  7. 2018Group Normalizationcited by 1,514
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  10. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  11. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  12. 2018LEAF: A Benchmark for Federated Settingscited by 780
20 more
  1. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  2. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  3. 2019Fair Resource Allocation in Federated Learningcited by 389
  4. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  5. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  6. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  7. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  8. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 202
  9. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  10. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 179
  11. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  12. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  13. 2019On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning Algorithmscited by 132
  14. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  15. 2014Analysis and Design of Optimization Algorithms via Integral Quadratic Constraintscited by 105
  16. 2020Federated Accelerated Stochastic Gradient Descentcited by 85
  17. 2020From Local SGD to Local Fixed-Point Methods for Federated Learningcited by 74
  18. 2019Provable Guarantees for Gradient-Based Meta-Learningcited by 67
  19. 2020Training Keyword Spotting Models on Non-IID Data with Federated Learningcited by 36
  20. 2019Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Ratescited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 8; 2023: 16; 2024: 7; 2025: 7; 2026: 5.

52 papers.

Sort:

By area:

  1. 2021A Field Guide to Federated Optimizationcited by 312
  2. 2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 108
  3. 2021On Large-Cohort Training for Federated Learningcited by 73
  4. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 55
  5. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  6. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  7. 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Datacited by 36
  8. 2021Local Adaptivity in Federated Learning: Convergence and Consistencycited by 31
  9. 2022Motley: Benchmarking Heterogeneity and Personalization in Federated Learningcited by 31
  10. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  11. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learningcited by 29
  12. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 25
38 more
  1. 2023Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initializationcited by 21
  2. 2021Exploiting Heterogeneity in Robust Federated Best-Arm Identificationcited by 20
  3. 2021A Non-parametric View of FedAvg and FedProx: Beyond Stationary Pointscited by 13
  4. 2022Federated Learning on Riemannian Manifoldscited by 13
  5. 2023Share Your Representation Only: Guaranteed Improvement of the Privacy-Utility Tradeoff in Federated Learningcited by 13
  6. 2021FLIX: A Simple and Communication-Efficient Alternative to Local Methods in Federated Learningcited by 12
  7. 2023Efficient Federated Prompt Tuning for Black-box Large Pre-trained Modelscited by 12
  8. 2022CompressedScaffnew: The First Theoretical Double Acceleration of Communication from Local Training and Compression in Distributed Optimizationcited by 11
  9. 2022FL_PyTorch: optimization research simulator for federated learningcited by 10
  10. 2023Combating Data Imbalances in Federated Semi-supervised Learning with Dual Regulatorscited by 9
  11. 2021Iterated Vector Fields and Conservatism, with Applications to Federated Learningcited by 8
  12. 2023Federated Learning with Manifold Regularization and Normalized Update Reaggregationcited by 8
  13. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environmentscited by 7
  14. 2025Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Accelerationcited by 7
  15. 2024FedCal: Achieving Local and Global Calibration in Federated Learning via Aggregated Parameterized Scalercited by 6
  16. 2022PersA-FL: Personalized Asynchronous Federated Learningcited by 5
  17. 2021Accelerating Federated Learning with a Global Biased Optimisercited by 4
  18. 2023Faster Federated Learning with Decaying Number of Local SGD Stepscited by 4
  19. 2023Leveraging Function Space Aggregation for Federated Learning at Scalecited by 4
  20. 2021An Operator Splitting View of Federated Learningcited by 2
  21. 2023Federated Automatic Differentiationcited by 2
  22. 2023FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Datacited by 2
  23. 2024A-FedPD: Aligning Dual-Drift is All Federated Primal-Dual Learning Needscited by 2
  24. 2023Federated Gradient Matching Pursuitcited by 1
  25. 2023SalientGrads: Sparse Models for Communication Efficient and Data Aware Distributed Federated Trainingcited by 1
  26. 2023On First-Order Meta-Reinforcement Learning with Moreau Envelopescited by 1
  27. 2024One-Shot Federated Learning with Bayesian Pseudocoresetscited by 1
  28. 2025Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Samplingcited by 1
  29. 2025CycleSL: Server-Client Cyclical Update Driven Scalable Split Learningcited by 1
  30. 2026Tight Analysis of Decentralized SGD: A Markov Chain Perspectivecited by 1
  31. 2026High dimensional theory of two-phase optimizerscited by 1
  32. 2024FedEGG: Federated Learning with Explicit Global Guidancecited by 0
  33. 2024Refined Analysis of Federated Averaging and Federated Richardson-Rombergcited by 0
  34. 2025Diffusion Model-Based Data Synthesis Aided Federated Semi-Supervised Learningcited by 0
  35. 2025Regularized Top-k: A Bayesian Framework for Gradient Sparsificationcited by 0
  36. 2025Drift-Aware Federated Learning: A Causal Perspectivecited by 0
  37. 2025Communication Efficient Federated Learning with Linear Convergence on Heterogeneous Datacited by 0
  38. 2026Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalizationcited by 0

50 most cited of 52 All 52 (JSON)