mlsanityData to 7 Oct 2026

On Second-order Optimization Methods for Federated Learning

arXiv 2109.02388 ↗First posted 6 Sep 2021cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

We consider federated learning (FL), where the training data is distributed across a large number of clients. The standard optimization method in this setting is Federated Averaging (FedAvg), which performs multiple local first-order optimization steps between communication rounds. In this work, we evaluate the performance of several second-order distributed methods with local steps in the FL setting which promise to have favorable convergence properties. We (i) show that FedAvg performs surprisingly well against its second-order competitors when evaluated under fair metrics (equal amount of local computations)-in contrast to the results of previous work. Based on our numerical study, we propose (ii) a novel variant that uses second-order local information for updates and a global line search to counteract the resulting local specificity.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 6 Sep 2021.

Builds on

20 of 36 references.

References · 20

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  5. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  6. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  7. 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 131
  8. 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
  9. 2020FedDANE: A Federated Newton-Type Methodcited by 76
  10. 2016CoCoA: A General Framework for Communication-Efficient Distributed Optimizationcited by 69
  11. 2021FedNL: Making Newton-Type Methods Applicable to Federated Learningcited by 50
  12. 2017GIANT: Globally Improved Approximate Newton Method for Distributed Optimizationcited by 43
8 more
  1. 2016AIDE: Fast and Communication Efficient Distributed Optimizationcited by 42
  2. 2021The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communicationcited by 30
  3. 2021Distributed Second Order Methods with Fast Rates and Compressed Communicationcited by 25
  4. 2019DINGO: Distributed Newton-Type Method for Gradient-Norm Optimizationcited by 20
  5. 2019Distributed estimation of the inverse Hessian by determinantal averagingcited by 16
  6. 2021LocalNewton: Reducing Communication Bottleneck for Distributed Learningcited by 15
  7. 2018A Distributed Second-Order Algorithm You Can Trustcited by 5
  8. 2020Distributed Adaptive Newton Methods with Global Superlinear Convergencecited by 4

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 3; 2025: 2.

9 papers.

  1. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  2. 2022Over-the-Air Federated Learning via Second-Order Optimizationcited by 11
  3. 2022FedSSO: A Federated Server-Side Second-Order Optimization Algorithmcited by 3
  4. 2024FAGH: Accelerating Federated Learning with Approximated Global Hessiancited by 1
  5. 2024Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learningcited by 1
  6. 2024Federated Learning with Uncertainty and Personalization via Efficient Second-order Optimizationcited by 1
  7. 2023Preconditioned Federated Learningcited by 0
  8. 2025Accelerated Training of Federated Learning via Second-Order Methodscited by 0
  9. 2025pFedSOP : Accelerating Training Of Personalized Federated Learning Using Second-Order Optimizationcited by 0