mlsanityData to 14 Oct 2026

On the Convergence of Local Descent Methods in Federated Learning

arXiv 1910.14425 ↗First posted 31 Oct 2019cs.LG cs.DC stat.MLcited by 153

Not on the map. Why?

Abstract (arXiv)

In federated distributed learning, the goal is to optimize a global training objective defined over distributed devices, where the data shard at each device is sampled from a possibly different distribution (a.k.a., heterogeneous or non i.i.d. data samples). In this paper, we generalize the local stochastic and full gradient descent with periodic averaging-- originally designed for homogeneous distributed optimization, to solve nonconvex optimization problems in federated learning. Although scant research is available on the effectiveness of local SGD in reducing the number of communication rounds in homogeneous setting, its convergence and communication complexity in heterogeneous setting is mostly demonstrated empirically and lacks through theoretical understating. To bridge this gap, we demonstrate that by properly analyzing the effect of unbiased gradients and sampling schema in federated setting, under mild assumptions, the implicit variance reduction feature of local distributed methods generalize to heterogeneous data shards and exhibits the best known convergence rates of homogeneous setting both in general nonconvex and under pl~ condition (generalization of strong-convexity). Our theoretical results complement the recent empirical studies that demonstrate the applicability of local GD/SGD to federated learning. We also specialize the proposed local method for networked distributed optimization. To the best of our knowledge, the obtained convergence rates are the sharpest known to date on the convergence of local decant methods with periodic averaging for solving nonconvex federated optimization in both centralized and networked distributed optimization.

Key facts. Cited by 153 papers in the map (data through 14 Oct 2026). First posted on arXiv 31 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 44; 2022: 36; 2023: 29; 2024: 20; 2025: 14; 2026: 10.

153 papers.

Sort:

By area:

  1. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  2. 2021Personalized Federated Learning using Hypernetworkscited by 212
  3. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  4. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
  5. 2021Multi-Center Federated Learning: Clients Clustering for Better Personalizationcited by 113
  6. 2021Distributionally Robust Federated Averagingcited by 104
  7. 2021Personalized Federated Learning with Gaussian Processescited by 76
  8. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  9. 2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 69
  10. 2021Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Samplingcited by 66
  11. 2021Semi-Decentralized Federated Learning with Cooperative D2D Local Model Aggregationscited by 51
  12. 2021Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiencycited by 40
38 more
  1. 2022No Free Lunch Theorem for Security and Utility in Federated Learningcited by 39
  2. 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Datacited by 38
  3. 2022Federated Minimax Optimization: Improved Convergence Analyses and Algorithmscited by 37
  4. 2021Minibatch vs Local SGD with Shuffling: Tight Convergence Bounds and Beyondcited by 36
  5. 2022Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspacescited by 36
  6. 2021Game of Gradients: Mitigating Irrelevant Clients in Federated Learningcited by 35
  7. 2021Personalized Federated Learning: A Unified Framework and Universal Optimization Techniquescited by 34
  8. 2022Variance Reduced ProxSkip: Algorithm, Theory and Application to Federated Learningcited by 33
  9. 2021FedJAX: Federated learning simulation with JAXcited by 32
  10. 2021Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learningcited by 31
  11. 2023On the Convergence of Federated Averaging with Cyclic Client Participationcited by 31
  12. 2021FedDR -- Randomized Douglas-Rachford Splitting Algorithms for Nonconvex Federated Composite Optimizationcited by 29
  13. 2021A New Look and Convergence Rate of Federated Multi-Task Learning with Laplacian Regularizationcited by 27
  14. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 27
  15. 2021Convergence Analysis and System Design for Federated Learning over Wireless Networkscited by 25
  16. 2022FedADMM: A Robust Federated Deep Learning Framework with Adaptivity to System Heterogeneitycited by 25
  17. 2021Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfercited by 23
  18. 2022Can 5th Generation Local Training Methods Support Client Sampling? Yes!cited by 22
  19. 2022FedShuffle: Recipes for Better Use of Local Work in Federated Learningcited by 21
  20. 2021FedPAGE: A Fast Local Stochastic Gradient Method for Communication-Efficient Federated Learningcited by 20
  21. 2021Exploiting Heterogeneity in Robust Federated Best-Arm Identificationcited by 20
  22. 2022Local Stochastic Bilevel Optimization with Momentum-Based Variance Reductioncited by 20
  23. 2022Walk for Learning: A Random Walk Approach for Federated Learning from Heterogeneous Datacited by 19
  24. 2022Communication Acceleration of Local Gradient Methods via an Accelerated Primal-Dual Algorithm with Inexact Proxcited by 19
  25. 2022GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexitycited by 18
  26. 2021FedDAG: Federated DAG Structure Learningcited by 17
  27. 2021A Decentralized Federated Learning Framework via Committee Mechanism with Convergence Guaranteecited by 16
  28. 2021FedFR: Joint Optimization Federated Framework for Generic and Personalized Face Recognitioncited by 16
  29. 2021On Bridging Generic and Personalized Federated Learning for Image Classificationcited by 15
  30. 2023Federated TD Learning over Finite-Rate Erasure Channels: Linear Speedup under Markovian Samplingcited by 15
  31. 2021Federated Learning from Small Datasetscited by 14
  32. 2021On Convergence of Federated Averaging Langevin Dynamicscited by 14
  33. 2022Anchor Sampling for Federated Learning with Partial Client Participationcited by 14
  34. 2022NET-FLEET: Achieving Linear Convergence Speedup for Fully Decentralized Federated Learning with Heterogeneous Datacited by 12
  35. 2021On the Convergence of Shallow Neural Network Training with Randomly Masked Neuronscited by 11
  36. 2022Communication-Efficient Robust Federated Learning with Noisy Labelscited by 11
  37. 2024Stable Diffusion-based Data Augmentation for Federated Learning with Non-IID Datacited by 11
  38. 2021Bandwidth Allocation for Multiple Federated Learning Services in Wireless Edge Networkscited by 10

50 most cited of 153 All 153 (JSON)