mlsanityData to 14 Oct 2026

Convergence Analysis of Federated Learning Methods Using Backward Error Analysis

arXiv 2503.03139 ↗First posted 5 Mar 2025cs.LG cs.AI cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Backward error analysis allows finding a modified loss function, which the parameter updates really follow under the influence of an optimization method. The additional loss terms included in this modified function is called implicit regularizer. In this paper, we attempt to find the implicit regularizer for various federated learning algorithms on non-IID data distribution, and explain why each method shows different convergence behavior. We first show that the implicit regularizer of FedAvg disperses the gradient of each client from the average gradient, thus increasing the gradient variance. We also empirically show that the implicit regularizer hampers its convergence. Similarly, we compute the implicit regularizers of FedSAM and SCAFFOLD, and explain why they converge better. While existing convergence analyses focus on pointing out the advantages of FedSAM and SCAFFOLD, our approach can explain their limitations in complex non-convex settings. In specific, we demonstrate that FedSAM can partially remove the bias in the first-order term of the implicit regularizer in FedAvg, whereas SCAFFOLD can fully eliminate the bias in the first-order term, but not in the second-order term. Consequently, the implicit regularizer can provide a useful insight on the convergence behavior of federated learning from a different theoretical perspective.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 31 references that are in the map. First posted on arXiv 5 Mar 2025.

Builds on

26 of 31 references.

References · 26

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  6. 2018Federated Learning with Non-IID Datacited by 1,108
  7. 2020Adaptive Federated Optimizationcited by 1,046
  8. 2018LEAF: A Benchmark for Federated Settingscited by 792
  9. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  10. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  11. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  12. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
14 more
  1. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  2. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  3. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 154
  4. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 153
  5. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  6. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  7. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  8. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  9. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
  10. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  11. 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
  12. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  13. 2023Why (and When) does Local SGD Generalize Better than SGD?cited by 26
  14. 2019Communication trade-offs for synchronized distributed SGD with large step sizecited by 4

Built on by

Not cited yet.