# Adaptive pruning-based Newton's method for distributed learning

arXiv: https://arxiv.org/abs/2308.10154 · HTML: https://mlsanity.com/paper/2308.10154/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 20 Aug 2023.

## Builds on (most cited first, 14 of 14)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [A Stochastic Quasi-Newton Method for Large-Scale Optimization](https://mlsanity.com/paper/1401.7020/) (2014)
- [FedNL: Making Newton-Type Methods Applicable to Federated Learning](https://mlsanity.com/paper/2106.02969/) (2021)
- [Distributed Second Order Methods with Fast Rates and Compressed Communication](https://mlsanity.com/paper/2102.07158/) (2021)
- [Distributed Learning of Deep Neural Networks using Independent Subnet Training](https://mlsanity.com/paper/1910.02120/) (2019)
- [On Variance Reduction in Stochastic Gradient Descent and its Asynchronous Variants](https://mlsanity.com/paper/1506.06840/) (2015)
- [Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learning](https://mlsanity.com/paper/2111.01847/) (2021)
- [Faster Stochastic Quasi-Newton Methods](https://mlsanity.com/paper/2004.06479/) (2020)
- [A Stochastic Newton Algorithm for Distributed Convex Optimization](https://mlsanity.com/paper/2110.02954/) (2021)
- [SAN: Stochastic Average Newton Algorithm for Minimizing Finite Sums](https://mlsanity.com/paper/2106.10520/) (2021)
- [Variance-Reduced Stochastic Quasi-Newton Methods for Decentralized Learning: Part I](https://mlsanity.com/paper/2201.07699/) (2022)
- [On the Convergence of Heterogeneous Federated Learning with Arbitrary Adaptive Online Model Pruning](https://mlsanity.com/paper/2201.11803/) (2022)
- [A Stochastic Second-Order Proximal Method for Distributed Optimization](https://mlsanity.com/paper/2211.10591/) (2022)

## Abstract (arXiv)

Newton's method leverages curvature information to boost performance, and thus outperforms first-order methods for distributed learning problems. However, Newton's method is not practical in large-scale and heterogeneous learning environments, due to obstacles such as high computation and communication costs of the Hessian matrix, sub-model diversity, staleness of training, and data heterogeneity. To overcome these obstacles, this paper presents a novel and efficient algorithm named Distributed Adaptive Newton Learning (textttDANL), which solves the drawbacks of Newton's method by using a simple Hessian initialization and adaptive allocation of training regions. The algorithm exhibits remarkable convergence properties, which are rigorously examined under standard assumptions in stochastic optimization. The theoretical analysis proves that textttDANL attains a linear convergence rate while efficiently adapting to available resources and keeping high efficiency. Furthermore, textttDANL shows notable independence from the condition number of the problem and removes the necessity for complex parameter tuning. Experiments demonstrate that textttDANL achieves linear convergence with efficient communication and strong performance across different datasets.