# Optimization of Graph Neural Networks with Natural Gradient Descent

arXiv: https://arxiv.org/abs/2008.09624 · HTML: https://mlsanity.com/paper/2008.09624/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2020.

## Built on by (most cited first, 26 of 26)

- [On Positional and Structural Node Features for Graph Neural Networks on Non-attributed Graphs](https://mlsanity.com/paper/2107.01495/) (2021)
- [Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures](https://mlsanity.com/paper/2311.00636/) (2023)
- [Training Data Attribution via Approximate Unrolled Differentiation](https://mlsanity.com/paper/2405.12186/) (2024)
- [The Neural Process Family: Survey, Applications and Perspectives](https://mlsanity.com/paper/2209.00517/) (2022)
- [ASDL: A Unified Interface for Gradient Preconditioning in PyTorch](https://mlsanity.com/paper/2305.04684/) (2023)
- [Geometry and convergence of natural policy gradient methods](https://mlsanity.com/paper/2211.02105/) (2022)
- [GCNBoost: Artwork Classification by Label Propagation through a Knowledge Graph](https://mlsanity.com/paper/2105.11852/) (2021)
- [Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC](https://mlsanity.com/paper/2312.05705/) (2023)
- [LOGIN: A Large Language Model Consulted Graph Neural Network Training Framework](https://mlsanity.com/paper/2405.13902/) (2024)
- [FSW-GNN: A Bi-Lipschitz WL-Equivalent Graph Neural Network](https://mlsanity.com/paper/2410.09118/) (2024)
- [Structural Node Embeddings with Homomorphism Counts](https://mlsanity.com/paper/2308.15283/) (2023)
- [Projected Stochastic Gradient Descent with Quantum Annealed Binary Gradients](https://mlsanity.com/paper/2310.15128/) (2023)
- [Spatio-temporal Modeling for Large-scale Vehicular Networks Using Graph Convolutional Networks](https://mlsanity.com/paper/2103.07636/) (2021)
- [Beyond the Known: Novel Class Discovery for Open-world Graph Learning](https://mlsanity.com/paper/2403.19907/) (2024)
- [A Benchmark for Fairness-Aware Graph Learning](https://mlsanity.com/paper/2407.12112/) (2024)
- [Graph Neural Networks Need Cluster-Normalize-Activate Modules](https://mlsanity.com/paper/2412.04064/) (2024)
- [An Energy-Based View of Graph Neural Networks](https://mlsanity.com/paper/2104.13492/) (2021)
- [Evaluating Neighbor Explainability for Graph Neural Networks](https://mlsanity.com/paper/2311.08118/) (2023)
- [ReFill: Reinforcement Learning for Fill-In Minimization](https://mlsanity.com/paper/2501.16130/) (2025)
- [Quadratic GCN for Graph Classification](https://mlsanity.com/paper/2104.06750/) (2021)
- [Affinity Mixup for Weakly Supervised Sound Event Detection](https://mlsanity.com/paper/2106.11233/) (2021)
- [Extrinsic Bayesian Optimizations on Manifolds](https://mlsanity.com/paper/2212.13886/) (2022)
- [GraphViz2Vec: A Structure-aware Feature Generation Model to Improve Classification in GNNs](https://mlsanity.com/paper/2401.17178/) (2024)
- [Conditional Temporal Neural Processes with Covariance Loss](https://mlsanity.com/paper/2504.00794/) (2025)
- [Visualization and Analysis of the Loss Landscape in Graph Neural Networks](https://mlsanity.com/paper/2509.11792/) (2025)
- [Multi-Scale Temporal Homeostasis Enables Efficient and Robust Neural Networks](https://mlsanity.com/paper/2602.07009/) (2026)

## Abstract (arXiv)

In this work, we propose to employ information-geometric tools to optimize a graph neural network architecture such as the graph convolutional networks. More specifically, we develop optimization algorithms for the graph-based semi-supervised learning by employing the natural gradient information in the optimization process. This allows us to efficiently exploit the geometry of the underlying statistical model or parameter space for optimization and inference. To the best of our knowledge, this is the first work that has utilized the natural gradient for the optimization of graph neural networks that can be extended to other semi-supervised problems. Efficient computations algorithms are developed and extensive numerical studies are conducted to demonstrate the superior performance of our algorithms over existing algorithms such as ADAM and SGD.