mlsanityData to 7 Oct 2026

SHED: A Newton-type algorithm for federated learning based on incremental Hessian eigenvector sharing

arXiv 2202.05800 ↗First posted 11 Feb 2022cs.LG cs.AI math.OCcited by 11

Not on the map. Why?

Abstract (arXiv)

There is a growing interest in the distributed optimization framework that goes under the name of Federated Learning (FL). In particular, much attention is being turned to FL scenarios where the network is strongly heterogeneous in terms of communication resources (e.g., bandwidth) and data distribution. In these cases, communication between local machines (agents) and the central server (Master) is a main consideration. In this work, we present SHED, an original communication-constrained Newton-type (NT) algorithm designed to accelerate FL in such heterogeneous scenarios. SHED is by design robust to non i.i.d. data distributions, handles heterogeneity of agents' communication resources (CRs), only requires sporadic Hessian computations, and achieves super-linear convergence. This is possible thanks to an incremental strategy, based on eigendecomposition of the local Hessian matrices, which exploits (possibly) outdated second-order information. The proposed solution is thoroughly validated on real datasets by assessing (i) the number of communication rounds required for convergence, (ii) the overall amount of data transmitted and (iii) the number of local Hessian computations. For all these metrics, the proposed approach shows superior performance against state-of-the art techniques like GIANT and FedNL.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 36 of its 59 references that are in the map. First posted on arXiv 11 Feb 2022.

Builds on

36 of 59 references.

References · 36

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2021Convex optimizationcited by 2,479
  6. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  7. 2018Federated Learning with Non-IID Datacited by 1,099
  8. 2017Federated Multi-Task Learningcited by 669
  9. 2020The Future of Digital Health with Federated Learningcited by 625
  10. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  11. 2021Federated Learning on Non-IID Data: A Surveycited by 345
  12. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
24 more
  1. 2019A Joint Learning and Communications Framework for Federated Learning over Wireless Networkscited by 262
  2. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  3. 2019Federated Learning over Wireless Fading Channelscited by 178
  4. 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 131
  5. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  6. 2018LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learningcited by 95
  7. 2020Communication-Efficient Edge AI: Algorithms and Systemscited by 95
  8. 2021FedNL: Making Newton-Type Methods Applicable to Federated Learningcited by 50
  9. 2017GIANT: Globally Improved Approximate Newton Method for Distributed Optimizationcited by 43
  10. 2016AIDE: Fast and Communication Efficient Distributed Optimizationcited by 42
  11. 2015Convergence rates of sub-sampled Newton methodscited by 32
  12. 2021Distributed Second Order Methods with Fast Rates and Compressed Communicationcited by 25
  13. 2019DINGO: Distributed Newton-Type Method for Gradient-Norm Optimizationcited by 20
  14. 2019A Survey of Multi-Access Edge Computing in 5G and Beyond: Fundamentals, Technology Integration, and State-of-the-Artcited by 18
  15. 2020Federated Learning under Channel Uncertainty: Joint Client Scheduling and Resource Allocationcited by 18
  16. 2019Distributed estimation of the inverse Hessian by determinantal averagingcited by 16
  17. 2021Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learningcited by 14
  18. 2021FedLess: Secure and Scalable Federated Learning Using Serverless Computingcited by 13
  19. 2021Resource-constrained Federated Edge Learning with Heterogeneous Data: Formulation and Analysiscited by 10
  20. 2021On the Convergence Time of Federated Learning Over Wireless Networks Under Imperfect CSIcited by 9
  21. 2020DONE: Distributed Approximate Newton-type Method for Federated Edge Learningcited by 8
  22. 2021Communication-Efficient Distributed Optimization with Quantized Preconditionerscited by 8
  23. 2019DAve-QN: A Distributed Averaged Quasi-Newton Method with Local Superlinear Convergence Ratecited by 5
  24. 2020Distributed Adaptive Newton Methods with Global Superlinear Convergencecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 3; 2025: 2; 2026: 2.

11 papers.

  1. 2022Distributed Newton-Type Methods with Communication Compression and Bernoulli Aggregationcited by 11
  2. 2023FedZeN: Towards superlinear zeroth-order federated learning via incremental Hessian estimationcited by 4
  3. 2025Distributed Quasi-Newton Method for Fair and Fast Federated Learningcited by 4
  4. 2024FedNS: A Fast Sketching Newton-Type Algorithm for Federated Learningcited by 3
  5. 2023Q-SHED: Distributed Optimization at the Edge via Hessian Eigenvectors Quantizationcited by 1
  6. 2024Federated Cubic Regularized Newton Learning with Sparsification-amplified Differential Privacycited by 1
  7. 2026Federated continual learning: A comprehensive survey on lifelong and privacy-preserving learning over distributed and non-stationary datacited by 1
  8. 2023VREM-FL: Mobility-Aware Computation-Scheduling Co-Design for Vehicular Federated Learningcited by 0
  9. 2024FLeNS: Federated Learning with Enhanced Nesterov-Newton Sketchcited by 0
  10. 2025Accelerated Training of Federated Learning via Second-Order Methodscited by 0
  11. 2026DP-FedSOFIM: Differentially Private Federated Stochastic Optimization using Regularized Fisher Information Matrixcited by 0