# Decentralized Riemannian natural gradient methods with Kronecker-product approximations

arXiv: https://arxiv.org/abs/2303.09611 · HTML: https://mlsanity.com/paper/2303.09611/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 46 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Builds on (most cited first, 22 of 22)

- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization](https://mlsanity.com/paper/1404.6264/) (2014)
- [On the Convergence of Decentralized Gradient Descent](https://mlsanity.com/paper/1310.7063/) (2013)
- [A Kronecker-factored approximate Fisher matrix for convolution layers](https://mlsanity.com/paper/1602.01407/) (2016)
- [Manopt, a Matlab toolbox for optimization on manifolds](https://mlsanity.com/paper/1308.5200/) (2013)
- [Global rates of convergence for nonconvex optimization on manifolds](https://mlsanity.com/paper/1605.08101/) (2016)
- [A decentralized proximal-gradient method with network independent step-sizes and separated convergence rates](https://mlsanity.com/paper/1704.07807/) (2017)
- [Riemannian approach to batch normalization](https://mlsanity.com/paper/1709.09603/) (2017)
- [A Brief Introduction to Manifold Optimization](https://mlsanity.com/paper/1906.05450/) (2019)
- [Decentralized Riemannian Gradient Descent on the Stiefel Manifold](https://mlsanity.com/paper/2102.07091/) (2021)
- [Distributed Nonconvex Constrained Optimization over Time-Varying Digraphs](https://mlsanity.com/paper/1809.01106/) (2018)
- [Exact Diffusion for Distributed Optimization and Learning --- Part II: Convergence Analysis](https://mlsanity.com/paper/1702.05142/) (2017)
- [Decentralized Optimization Over the Stiefel Manifold by an Approximate Augmented Lagrangian Function](https://mlsanity.com/paper/2112.14949/) (2021)
- [On the Local Linear Rate of Consensus on the Stiefel Manifold](https://mlsanity.com/paper/2101.09346/) (2021)
- [Distributed Optimization on Riemannian Manifolds for multi-agent networks](https://mlsanity.com/paper/1711.11196/) (2017)
- [Riemannian Natural Gradient Methods](https://mlsanity.com/paper/2207.07287/) (2022)
- [Decentralized Quasi-Newton Methods](https://mlsanity.com/paper/1605.00933/) (2016)
- [Distributed Adaptive Newton Methods with Global Superlinear Convergence](https://mlsanity.com/paper/2002.07378/) (2020)
- [Variance-Reduced Stochastic Quasi-Newton Methods for Decentralized Learning: Part I](https://mlsanity.com/paper/2201.07699/) (2022)
- [DQM: Decentralized Quadratically Approximated Alternating Direction Method of Multipliers](https://mlsanity.com/paper/1508.02073/) (2015)
- [Decentralized Quadratically Approximated Alternating Direction Method of Multipliers](https://mlsanity.com/paper/1510.07356/) (2015)

## Built on by (most cited first, 1 of 1)

- [Decentralized Optimization on Compact Submanifolds by Quantized Riemannian Gradient Tracking](https://mlsanity.com/paper/2506.07351/) (2025)

## Abstract (arXiv)

With a computationally efficient approximation of the second-order information, natural gradient methods have been successful in solving large-scale structured optimization problems. We study the natural gradient methods for the large-scale decentralized optimization problems on Riemannian manifolds, where the local objective function defined by the local dataset is of a log-probability type. By utilizing the structure of the Riemannian Fisher information matrix (RFIM), we present an efficient decentralized Riemannian natural gradient descent (DRNGD) method. To overcome the communication issue of the high-dimension RFIM, we consider a class of structured problems for which the RFIM can be approximated by a Kronecker product of two low-dimension matrices. By performing the communications over the Kronecker factors, a high-quality approximation of the RFIM can be obtained in a low cost. We prove that DRNGD converges to a stationary point with the best-known rate of O(1/K). Numerical experiments demonstrate the efficiency of our proposed method compared with the state-of-the-art ones. To the best of our knowledge, this is the first Riemannian second-order method for solving decentralized manifold optimization problems.