# Riemannian metrics for neural networks I: feedforward networks

arXiv: https://arxiv.org/abs/1303.0818 · HTML: https://mlsanity.com/paper/1303.0818/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2013.

## Built on by (most cited first, 30 of 30)

- [Input Similarity from the Neural Network Perspective](https://mlsanity.com/paper/2102.05262/) (2021)
- [Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks](https://mlsanity.com/paper/2105.02716/) (2021)
- [Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization](https://mlsanity.com/paper/2201.12250/) (2022)
- [On the Variance of the Fisher Information for Deep Learning](https://mlsanity.com/paper/2107.04205/) (2021)
- [ASDL: A Unified Interface for Gradient Preconditioning in PyTorch](https://mlsanity.com/paper/2305.04684/) (2023)
- [Wasserstein Proximal of GANs](https://mlsanity.com/paper/2102.06862/) (2021)
- [Invariance Properties of the Natural Gradient in Overparametrised Systems](https://mlsanity.com/paper/2206.15273/) (2022)
- [On a continuous time model of gradient descent dynamics and instability in deep learning](https://mlsanity.com/paper/2302.01952/) (2023)
- [Trade-Offs of Diagonal Fisher Information Matrix Estimators](https://mlsanity.com/paper/2402.05379/) (2024)
- [Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decomposition](https://mlsanity.com/paper/2201.10285/) (2022)
- [A Mini-Block Fisher Method for Deep Neural Networks](https://mlsanity.com/paper/2202.04124/) (2022)
- [Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling](https://mlsanity.com/paper/2410.15143/) (2024)
- [ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning](https://mlsanity.com/paper/2412.10782/) (2024)
- [ISAAC Newton: Input-based Approximate Curvature for Newton's Method](https://mlsanity.com/paper/2305.00604/) (2023)
- [Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models](https://mlsanity.com/paper/2410.00131/) (2024)
- [On the Interpretability of Attention Networks](https://mlsanity.com/paper/2212.14776/) (2022)
- [Generalized Tangent Kernel: A Unified Geometric Foundation for Natural Gradient and Standard Gradient](https://mlsanity.com/paper/2202.06232/) (2022)
- [Information Geometry of Dropout Training](https://mlsanity.com/paper/2206.10936/) (2022)
- [Fisher Width: A Geometric Measure of Complexity on Statistical Manifolds](https://mlsanity.com/paper/2606.18306/) (2026)
- [Gradient representations in ReLU networks as similarity functions](https://mlsanity.com/paper/2110.13581/) (2021)
- [On the Natural Gradient of the Evidence Lower Bound](https://mlsanity.com/paper/2307.11249/) (2023)
- [Gathering and Exploiting Higher-Order Information when Training Large Structured Models](https://mlsanity.com/paper/2312.03885/) (2023)
- [The Geometry of Projection Heads: Conditioning, Invariance, and Collapse](https://mlsanity.com/paper/2605.17180/) (2026)
- [Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss](https://mlsanity.com/paper/2606.06418/) (2026)
- [Component-Wise Natural Gradient Descent -- An Efficient Neural Network Optimization](https://mlsanity.com/paper/2210.05268/) (2022)
- [Analysis and Comparison of Two-Level KFAC Methods for Training Deep Neural Networks](https://mlsanity.com/paper/2303.18083/) (2023)
- [Neural Harmonium: An Interpretable Deep Structure for Nonlinear Dynamic System Identification with Application to Audio Processing](https://mlsanity.com/paper/2310.07032/) (2023)
- [An extended asymmetric sigmoid with Perceptron (SIGTRON) for imbalanced linear classification](https://mlsanity.com/paper/2312.16043/) (2023)
- [Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifolds](https://mlsanity.com/paper/2505.13614/) (2025)
- [Canonical Regularisation of Wide Feature-Learning Neural Networks](https://mlsanity.com/paper/2605.18180/) (2026)

## Abstract (arXiv)

We describe four algorithms for neural network training, each adapted to different scalability constraints. These algorithms are mathematically principled and invariant under a number of transformations in data and network representation, from which performance is thus independent. These algorithms are obtained from the setting of differential geometry, and are based on either the natural gradient using the Fisher information matrix, or on Hessian methods, scaled down in a specific way to allow for scalability while keeping some of their key mathematical properties.