mlsanityData to 14 Oct 2026

On the Outsized Importance of Learning Rates in Local Update Methods

arXiv 2007.00878 ↗First posted 2 Jul 2020cs.LG math.OC stat.MLcited by 36

Not on the map. Why?

Abstract (arXiv)

We study a family of algorithms, which we refer to as local update methods, that generalize many federated learning and meta-learning algorithms. We prove that for quadratic objectives, local update methods perform stochastic gradient descent on a surrogate loss function which we exactly characterize. We show that the choice of client learning rate controls the condition number of that surrogate loss, as well as the distance between the minimizers of the surrogate and true loss functions. We use this theory to derive novel convergence rates for federated averaging that showcase this trade-off between the condition number of the surrogate loss and its alignment with the true loss function. We validate our results empirically, showing that in communication-limited settings, proper learning rate tuning is often sufficient to reach near-optimal behavior. We also present a practical method for automatic learning rate decay in local update methods that helps reduce the need for learning rate tuning, and highlight its empirical performance on a variety of tasks and datasets.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). First posted on arXiv 2 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 14; 2022: 5; 2023: 7; 2024: 5; 2025: 3; 2026: 2.

36 papers.

Sort:

  1. 2021A Field Guide to Federated Optimizationcited by 315
  2. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
  3. 2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 69
  4. 2021What Do We Mean by Generalization in Federated Learning?cited by 62
  5. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 56
  6. 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
  7. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 27
  8. 2021Exploiting Heterogeneity in Robust Federated Best-Arm Identificationcited by 20
  9. 2021CodedPaddedFL and CodedSecAgg: Straggler Mitigation and Secure Aggregation in Federated Learningcited by 20
  10. 2023Federated Learning as Variational Inference: A Scalable Expectation Propagation Approachcited by 17
  11. 2021On Data Efficiency of Meta-learningcited by 16
  12. 2023Federated TD Learning over Finite-Rate Erasure Channels: Linear Speedup under Markovian Samplingcited by 15
24 more
  1. 2021On Convergence of Federated Averaging Langevin Dynamicscited by 14
  2. 2021A Non-parametric View of FedAvg and FedProx: Beyond Stationary Pointscited by 13
  3. 2024Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptationcited by 13
  4. 2024The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communicationcited by 12
  5. 2024Task Arithmetic Through The Lens Of One-Shot Federated Learningcited by 12
  6. 2021FedChain: Chained Algorithms for Near-Optimal Communication Cost in Federated Learningcited by 11
  7. 2023Over-the-Air Federated Learning and Optimizationcited by 10
  8. 2025Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Accelerationcited by 9
  9. 2021A Marketplace for Trading AI Models based on Blockchain and Incentives for IoT Datacited by 8
  10. 2023Improving Accelerated Federated Learning with Compression and Importance Samplingcited by 8
  11. 2021Adaptive Client Sampling in Federated Learning via Online Learning with Bandit Feedbackcited by 7
  12. 2022FedNST: Federated Noisy Student Training for Automatic Speech Recognitioncited by 5
  13. 2021Coding for Straggler Mitigation in Federated Learningcited by 4
  14. 2022Straggler-Resilient Differentially-Private Decentralized Learningcited by 3
  15. 2022Online Meta-Learning for Model Update Aggregation in Federated Learning for Click-Through Rate Predictioncited by 2
  16. 2022Federated Hypergradient Descentcited by 2
  17. 2024FedECADO: A Dynamical System Model of Federated Learningcited by 2
  18. 2023SalientGrads: Sparse Models for Communication Efficient and Data Aware Distributed Federated Trainingcited by 1
  19. 2025First Provable Guarantees for Practical Private FL: Beyond Restrictive Assumptionscited by 1
  20. 2021Boosting Resource-Constrained Federated Learning Systems with Guessed Updatescited by 0
  21. 2024Approximate Gradient Coding for Privacy-Flexible Federated Learning with Non-IID Datacited by 0
  22. 2025Adaptive Federated Learning via Dynamical System Modelcited by 0
  23. 2026What's in a Smoothness Constant? Tighter Rates for Local SGD with Bounded Second-order Heterogeneitycited by 0
  24. 2026Theoretical Foundations of Communication-Efficient, Robust, and Practical Distributed and Federated Optimizationcited by 0