mlsanityData to 14 Oct 2026

Preconditioned Federated Learning

arXiv 2309.11378 ↗First posted 20 Sep 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) is a distributed machine learning approach that enables model training in communication efficient and privacy-preserving manner. The standard optimization method in FL is Federated Averaging (FedAvg), which performs multiple local SGD steps between communication rounds. FedAvg has been considered to lack algorithm adaptivity compared to modern first-order adaptive optimizations. In this paper, we propose new communication-efficient FL algortithms based on two adaptive frameworks: local adaptivity (PreFed) and server-side adaptivity (PreFedOp). Proposed methods adopt adaptivity by using a novel covariance matrix preconditioner. Theoretically, we provide convergence guarantees for our algorithms. The empirical experiments show our methods achieve state-of-the-art performances on both i.i.d. and non-i.i.d. settings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 29 references that are in the map. First posted on arXiv 20 Sep 2023.

Builds on

22 of 29 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  7. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  8. 2020Adaptive Federated Optimizationcited by 1,046
  9. 2019On the Convergence of Adam and Beyondcited by 841
  10. 2016Learning to learn by gradient descent by gradient descentcited by 620
  11. 2014The Loss Surfaces of Multilayer Networkscited by 280
  12. 2018AdaGrad stepsizes: Sharp convergence over nonconvex landscapescited by 182
10 more
  1. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 175
  2. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 153
  3. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  4. 2017Noisy Natural Gradient as Variational Inferencecited by 94
  5. 2015Equilibrated adaptive learning rates for non-convex optimizationcited by 42
  6. 2019Why are Adaptive Methods Good for Attention Models?cited by 35
  7. 2019Escaping Saddle Points with Adaptive Gradient Methodscited by 34
  8. 2019Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Ratescited by 26
  9. 2018Preconditioner on Matrix Lie Group for SGDcited by 16
  10. 2021On Second-order Optimization Methods for Federated Learningcited by 9

Built on by

Not cited yet.