Accelerated Training of Federated Learning via Second-Order Methods
Not on the map. Why?
Abstract (arXiv)
This paper explores second-order optimization methods in Federated Learning (FL), addressing the critical challenges of slow convergence and the excessive communication rounds required to achieve optimal performance from the global model. While existing surveys in FL primarily focus on challenges related to statistical and device label heterogeneity, as well as privacy and security concerns in first-order FL methods, less attention has been given to the issue of slow model training. This slow training often leads to the need for excessive communication rounds or increased communication costs, particularly when data across clients are highly heterogeneous. In this paper, we examine various FL methods that leverage second-order optimization to accelerate the training process. We provide a comprehensive categorization of state-of-the-art second-order FL methods and compare their performance based on convergence speed, computational cost, memory usage, transmission overhead, and generalization of the global model. Our findings show the potential of incorporating Hessian curvature through second-order optimization into FL and highlight key challenges, such as the efficient utilization of Hessian and its inverse in FL. This work lays the groundwork for future research aimed at developing scalable and efficient federated optimization methods for improving the training of the global model in FL.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 55 of its 102 references that are in the map. First posted on arXiv 29 May 2025.
Builds on
55 of 102 references.
References · 55
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018Federated Optimization in Heterogeneous Networkscited by 2,987
- 2018Federated Learning with Non-IID Datacited by 1,108
- 2020Adaptive Federated Optimizationcited by 1,046
- 2018LEAF: A Benchmark for Federated Settingscited by 792
- 2021Model-Contrastive Federated Learningcited by 766
- 2020Federated Learning with Matched Averagingcited by 564
- 2021Federated Learning Based on Dynamic Regularizationcited by 562
- 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 536
- 2021Federated Learning on Non-IID Data: A Surveycited by 346
- 2020Attack of the Tails: Yes, You Really Can Backdoor Federated Learningcited by 309
- 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
38 more
- 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
- 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 132
- 2020LDP-Fed: Federated Learning with Local Differential Privacycited by 132
- 2019Overcoming Forgetting in Federated Learning on Non-IID Datacited by 120
- 2019Distributed Learning with Compressed Gradient Differencescited by 112
- 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 99
- 2020Acceleration for Compressed Gradient Descent in Distributed and Federated Optimizationcited by 90
- 2020Local SGD: Unified Theory and New Efficient Methodscited by 88
- 2020Federated Accelerated Stochastic Gradient Descentcited by 85
- 2020FedDANE: A Federated Newton-Type Methodcited by 76
- 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
- 2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 69
- 2022FedVARP: Tackling the Variance Due to Partial Client Participation in Federated Learningcited by 64
- 2021Federated Learning with Fair Averagingcited by 61
- 2019Federated Learning with Bayesian Differential Privacycited by 60
- 2015A Universal Catalyst for First-Order Optimizationcited by 54
- 2021FedNL: Making Newton-Type Methods Applicable to Federated Learningcited by 52
- 2017GIANT: Globally Improved Approximate Newton Method for Distributed Optimizationcited by 44
- 2016AIDE: Fast and Communication Efficient Distributed Optimizationcited by 43
- 2021Implicit Gradient Alignment in Distributed and Federated Learningcited by 29
- 2021Distributed Second Order Methods with Fast Rates and Compressed Communicationcited by 25
- 2015Distributed Optimization with Arbitrary Local Solverscited by 23
- 2022FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learningcited by 23
- 2023Understanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initializationcited by 21
- 2019DINGO: Distributed Newton-Type Method for Gradient-Norm Optimizationcited by 20
- 2019Distributed estimation of the inverse Hessian by determinantal averagingcited by 16
- 2021Differential Privacy Meets Federated Learning under Communication Constraintscited by 16
- 2021LocalNewton: Reducing Communication Bottleneck for Distributed Learningcited by 15
- 2021Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learningcited by 15
- 2022Anchor Sampling for Federated Learning with Partial Client Participationcited by 14
- 2022SHED: A Newton-type algorithm for federated learning based on incremental Hessian eigenvector sharingcited by 11
- 2022Over-the-Air Federated Learning via Second-Order Optimizationcited by 11
- 2021On Second-order Optimization Methods for Federated Learningcited by 9
- 2020DONE: Distributed Approximate Newton-type Method for Federated Edge Learningcited by 8
- 2023Exploring adversarial attacks in federated learning for medical imagingcited by 8
- 2015Hessian-free Optimization for Learning Deep Multidimensional Recurrent Neural Networkscited by 5
- 2024Fed-Sophia: A Communication-Efficient Second-Order Federated Learning Algorithmcited by 5
- 2025Distributed Quasi-Newton Method for Fair and Fast Federated Learningcited by 5
50 most cited of 55 All 55 (JSON)
Built on by
Not cited yet.