mlsanityData to 7 Oct 2026

Local Adaptivity in Federated Learning: Convergence and Consistency

arXiv 2106.02305 ↗First posted 4 Jun 2021cs.LG cs.DC stat.MLcited by 31

Not on the map. Why?

Abstract (arXiv)

The federated learning (FL) framework trains a machine learning model using decentralized data stored at edge client devices by periodically aggregating locally trained models. Popular optimization algorithms of FL use vanilla (stochastic) gradient descent for both local updates at clients and global updates at the aggregating server. Recently, adaptive optimization methods such as AdaGrad have been studied for server updates. However, the effect of using adaptive optimization methods for local updates at clients is not yet understood. We show in both theory and practice that while local adaptive methods can accelerate convergence, they can cause a non-vanishing solution bias, where the final converged solution may be different from the stationary point of the global objective function. We propose correction techniques to overcome this inconsistency and complement the local adaptive methods for FL. Extensive experiments on realistic federated training tasks show that the proposed algorithms can achieve faster convergence and higher test accuracy than the baselines without local adaptivity.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 31 of its 39 references that are in the map. First posted on arXiv 4 Jun 2021.

Builds on

31 of 39 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  6. 2020Adaptive Federated Optimizationcited by 1,036
  7. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  8. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  9. 2019On the Convergence of Adam and Beyondcited by 834
  10. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  11. 2014New insights and perspectives on the natural gradient methodcited by 435
  12. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
19 more
  1. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  2. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  3. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 196
  4. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 178
  5. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  6. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  7. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
  8. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 143
  9. 2020Optimal Client Sampling for Federated Learningcited by 123
  10. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  11. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  12. 2019Unified Optimal Analysis of the (Stochastic) Gradient Methodcited by 81
  13. 2019Variance Reduced Local SGD with Lower Communication Complexitycited by 80
  14. 2020FedDANE: A Federated Newton-Type Methodcited by 76
  15. 2020From Local SGD to Local Fixed-Point Methods for Federated Learningcited by 74
  16. 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
  17. 2020Communication-Efficient Federated Learning via Optimal Client Samplingcited by 39
  18. 2019Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Ratescited by 26
  19. 2020Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Datacited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 11; 2024: 9; 2025: 3; 2026: 2.

31 papers.

Sort:

  1. 2023Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimizationcited by 65
  2. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 55
  3. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  4. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  5. 2022Accelerated Federated Learning with Decoupled Adaptive Optimizationcited by 23
  6. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  7. 2023Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initializationcited by 21
  8. 2021Federated Learning for Internet of Things: A Federated Learning Framework for On-device Anomaly Data Detectioncited by 19
  9. 2022FedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregationcited by 16
  10. 2024Federated Learning in Practice: Reflections and Projectionscited by 15
  11. 2024Federated Learning with Bilateral Curation for Partially Class-Disjoint Datacited by 13
  12. 2023Adaptive Federated Learning with Auto-Tuned Clientscited by 7
19 more
  1. 2023Locally Adaptive Federated Learningcited by 7
  2. 2024TurboSVM-FL: Boosting Federated Learning through SVM Aggregation for Lazy Clientscited by 7
  3. 2025Efficient Distributed Optimization under Heavy-Tailed Noisecited by 7
  4. 2022Addressing Heterogeneity in Federated Learning via Distributional Transformationcited by 6
  5. 2024Efficient Federated Learning Using Dynamic Update and Adaptive Pruning with Momentum on Shared Server Datacited by 6
  6. 2023Memory-adaptive Depth-wise Heterogeneous Federated Learningcited by 5
  7. 2024Federated brain tumor segmentation: an extensive benchmarkcited by 5
  8. 2025Layerwise Federated Learning for Heterogeneous Quantum Clients using Quoruscited by 4
  9. 2023Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Datacited by 3
  10. 2023FedDA: Faster Framework of Local Adaptive Gradient Methods via Restarted Dual Averagingcited by 2
  11. 2023FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Datacited by 2
  12. 2024Efficient Adaptive Federated Optimizationcited by 2
  13. 2024Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learningcited by 1
  14. 2026Tight Analysis of Decentralized SGD: A Markov Chain Perspectivecited by 1
  15. 2022On the Stability Analysis of Open Federated Learning Systemscited by 0
  16. 2024FedEGG: Federated Learning with Explicit Global Guidancecited by 0
  17. 2024Refined Analysis of Federated Averaging and Federated Richardson-Rombergcited by 0
  18. 2025FedDuA: Doubly Adaptive Federated Learningcited by 0
  19. 2026Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalizationcited by 0