# SHED: A Newton-type algorithm for federated learning based on incremental Hessian eigenvector sharing arXiv: https://arxiv.org/abs/2202.05800 ยท HTML: https://mlsanity.com/paper/2202.05800/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 36 of its 59 references that are in the map. - First posted on arXiv 11 Feb 2022. ## Builds on (most cited first, 30 of 36) - [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016) - [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017) - [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019) - [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018) - [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021) - [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019) - [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018) - [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017) - [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020) - [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018) - [Federated Learning on Non-IID Data: A Survey](https://mlsanity.com/paper/2106.06843/) (2021) - [Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning](https://mlsanity.com/paper/1807.06629/) (2018) - [A Joint Learning and Communications Framework for Federated Learning over Wireless Networks](https://mlsanity.com/paper/1909.07972/) (2019) - [Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data](https://mlsanity.com/paper/1811.11479/) (2018) - [Federated Learning over Wireless Fading Channels](https://mlsanity.com/paper/1907.09769/) (2019) - [Communication Efficient Distributed Optimization using an Approximate Newton-type Method](https://mlsanity.com/paper/1312.7853/) (2013) - [Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradients](https://mlsanity.com/paper/2102.07053/) (2021) - [LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning](https://mlsanity.com/paper/1805.09965/) (2018) - [Communication-Efficient Edge AI: Algorithms and Systems](https://mlsanity.com/paper/2002.09668/) (2020) - [FedNL: Making Newton-Type Methods Applicable to Federated Learning](https://mlsanity.com/paper/2106.02969/) (2021) - [GIANT: Globally Improved Approximate Newton Method for Distributed Optimization](https://mlsanity.com/paper/1709.03528/) (2017) - [AIDE: Fast and Communication Efficient Distributed Optimization](https://mlsanity.com/paper/1608.06879/) (2016) - [Convergence rates of sub-sampled Newton methods](https://mlsanity.com/paper/1508.02810/) (2015) - [Distributed Second Order Methods with Fast Rates and Compressed Communication](https://mlsanity.com/paper/2102.07158/) (2021) - [DINGO: Distributed Newton-Type Method for Gradient-Norm Optimization](https://mlsanity.com/paper/1901.05134/) (2019) - [A Survey of Multi-Access Edge Computing in 5G and Beyond: Fundamentals, Technology Integration, and State-of-the-Art](https://mlsanity.com/paper/1906.08452/) (2019) - [Federated Learning under Channel Uncertainty: Joint Client Scheduling and Resource Allocation](https://mlsanity.com/paper/2002.00802/) (2020) - [Distributed estimation of the inverse Hessian by determinantal averaging](https://mlsanity.com/paper/1905.11546/) (2019) - [Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learning](https://mlsanity.com/paper/2111.01847/) (2021) - [FedLess: Secure and Scalable Federated Learning Using Serverless Computing](https://mlsanity.com/paper/2111.03396/) (2021) ## Built on by (most cited first, 11 of 11) - [Distributed Newton-Type Methods with Communication Compression and Bernoulli Aggregation](https://mlsanity.com/paper/2206.03588/) (2022) - [Distributed Quasi-Newton Method for Fair and Fast Federated Learning](https://mlsanity.com/paper/2501.10877/) (2025) - [FedZeN: Towards superlinear zeroth-order federated learning via incremental Hessian estimation](https://mlsanity.com/paper/2309.17174/) (2023) - [FedNS: A Fast Sketching Newton-Type Algorithm for Federated Learning](https://mlsanity.com/paper/2401.02734/) (2024) - [Q-SHED: Distributed Optimization at the Edge via Hessian Eigenvectors Quantization](https://mlsanity.com/paper/2305.10852/) (2023) - [Federated Cubic Regularized Newton Learning with Sparsification-amplified Differential Privacy](https://mlsanity.com/paper/2408.04315/) (2024) - [Federated continual learning: A comprehensive survey on lifelong and privacy-preserving learning over distributed and non-stationary data](https://mlsanity.com/paper/2606.11272/) (2026) - [VREM-FL: Mobility-Aware Computation-Scheduling Co-Design for Vehicular Federated Learning](https://mlsanity.com/paper/2311.18741/) (2023) - [FLeNS: Federated Learning with Enhanced Nesterov-Newton Sketch](https://mlsanity.com/paper/2409.15216/) (2024) - [Accelerated Training of Federated Learning via Second-Order Methods](https://mlsanity.com/paper/2505.23588/) (2025) - [DP-FedSOFIM: Differentially Private Federated Stochastic Optimization using Regularized Fisher Information Matrix](https://mlsanity.com/paper/2601.09166/) (2026) ## Abstract (arXiv) There is a growing interest in the distributed optimization framework that goes under the name of Federated Learning (FL). In particular, much attention is being turned to FL scenarios where the network is strongly heterogeneous in terms of communication resources (e.g., bandwidth) and data distribution. In these cases, communication between local machines (agents) and the central server (Master) is a main consideration. In this work, we present SHED, an original communication-constrained Newton-type (NT) algorithm designed to accelerate FL in such heterogeneous scenarios. SHED is by design robust to non i.i.d. data distributions, handles heterogeneity of agents' communication resources (CRs), only requires sporadic Hessian computations, and achieves super-linear convergence. This is possible thanks to an incremental strategy, based on eigendecomposition of the local Hessian matrices, which exploits (possibly) outdated second-order information. The proposed solution is thoroughly validated on real datasets by assessing (i) the number of communication rounds required for convergence, (ii) the overall amount of data transmitted and (iii) the number of local Hessian computations. For all these metrics, the proposed approach shows superior performance against state-of-the art techniques like GIANT and FedNL.