# Network-GIANT: Fully distributed Newton-type optimization via harmonic Hessian consensus

arXiv: https://arxiv.org/abs/2305.07898 · HTML: https://mlsanity.com/paper/2305.07898/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 25 references that are in the map.
- First posted on arXiv 13 May 2023.

## Builds on (most cited first, 14 of 14)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [On the Convergence of Decentralized Gradient Descent](https://mlsanity.com/paper/1310.7063/) (2013)
- [Achieving Geometric Convergence for Distributed Optimization over Time-Varying Graphs](https://mlsanity.com/paper/1607.03218/) (2016)
- [Communication Efficient Distributed Optimization using an Approximate Newton-type Method](https://mlsanity.com/paper/1312.7853/) (2013)
- [On the Linear Convergence of the ADMM in Decentralized Consensus Optimization](https://mlsanity.com/paper/1307.5561/) (2013)
- [FedNL: Making Newton-Type Methods Applicable to Federated Learning](https://mlsanity.com/paper/2106.02969/) (2021)
- [GIANT: Globally Improved Approximate Newton Method for Distributed Optimization](https://mlsanity.com/paper/1709.03528/) (2017)
- [Lyapunov Theory for Discrete Time Systems](https://mlsanity.com/paper/1809.05289/) (2018)
- [DONE: Distributed Approximate Newton-type Method for Federated Edge Learning](https://mlsanity.com/paper/2012.05625/) (2020)
- [A Decentralized Second-Order Method with Exact Linear Convergence Rate for Consensus Optimization](https://mlsanity.com/paper/1602.00596/) (2016)
- [Newton-like method with diagonal correction for distributed optimization](https://mlsanity.com/paper/1509.01703/) (2015)
- [Newton-Raphson Consensus for Distributed Convex Optimization](https://mlsanity.com/paper/1511.01509/) (2015)
- [ZO-JADE: Zeroth-order Curvature-Aware Multi-Agent Convex Optimization](https://mlsanity.com/paper/2303.07450/) (2023)

## Built on by (most cited first, 1 of 1)

- [Federated Cubic Regularized Newton Learning with Sparsification-amplified Differential Privacy](https://mlsanity.com/paper/2408.04315/) (2024)

## Abstract (arXiv)

This paper considers the problem of distributed multi-agent learning, where the global aim is to minimize a sum of local objective (empirical loss) functions through local optimization and information exchange between neighbouring nodes. We introduce a Newton-type fully distributed optimization algorithm, Network-GIANT, which is based on GIANT, a Federated learning algorithm that relies on a centralized parameter server. The Network-GIANT algorithm is designed via a combination of gradient-tracking and a Newton-type iterative algorithm at each node with consensus based averaging of local gradient and Newton updates. We prove that our algorithm guarantees semi-global and exponential convergence to the exact solution over the network assuming strongly convex and smooth loss functions. We provide empirical evidence of the superior convergence performance of Network-GIANT over other state-of-art distributed learning algorithms such as Network-DANE and Newton-Raphson Consensus.