mlsanityData to 14 Oct 2026

Riemannian metrics for neural networks I: feedforward networks

arXiv 1303.0818 ↗First posted 4 Mar 2013cs.NE cs.IT cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

We describe four algorithms for neural network training, each adapted to different scalability constraints. These algorithms are mathematically principled and invariant under a number of transformations in data and network representation, from which performance is thus independent. These algorithms are obtained from the setting of differential geometry, and are based on either the natural gradient using the Fisher information matrix, or on Hessian methods, scaled down in a specific way to allow for scalability while keeping some of their key mathematical properties.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Mar 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 8; 2023: 8; 2024: 4; 2025: 1; 2026: 4.

30 papers.

Sort:

  1. 2021Input Similarity from the Neural Network Perspectivecited by 79
  2. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  3. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 30
  4. 2021On the Variance of the Fisher Information for Deep Learningcited by 22
  5. 2023ASDL: A Unified Interface for Gradient Preconditioning in PyTorchcited by 21
  6. 2021Wasserstein Proximal of GANscited by 15
  7. 2022Invariance Properties of the Natural Gradient in Overparametrised Systemscited by 11
  8. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
  9. 2024Trade-Offs of Diagonal Fisher Information Matrix Estimatorscited by 10
  10. 2022Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decompositioncited by 9
  11. 2022A Mini-Block Fisher Method for Deep Neural Networkscited by 8
  12. 2024Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Samplingcited by 7
18 more
  1. 2024ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learningcited by 7
  2. 2023ISAAC Newton: Input-based Approximate Curvature for Newton's Methodcited by 6
  3. 2024Fisher Information-based Efficient Curriculum Federated Learning with Large Language Modelscited by 6
  4. 2022On the Interpretability of Attention Networkscited by 5
  5. 2022Generalized Tangent Kernel: A Unified Geometric Foundation for Natural Gradient and Standard Gradientcited by 2
  6. 2022Information Geometry of Dropout Trainingcited by 2
  7. 2026Fisher Width: A Geometric Measure of Complexity on Statistical Manifoldscited by 2
  8. 2021Gradient representations in ReLU networks as similarity functionscited by 1
  9. 2023On the Natural Gradient of the Evidence Lower Boundcited by 1
  10. 2023Gathering and Exploiting Higher-Order Information when Training Large Structured Modelscited by 1
  11. 2026The Geometry of Projection Heads: Conditioning, Invariance, and Collapsecited by 1
  12. 2026Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Losscited by 1
  13. 2022Component-Wise Natural Gradient Descent -- An Efficient Neural Network Optimizationcited by 0
  14. 2023Analysis and Comparison of Two-Level KFAC Methods for Training Deep Neural Networkscited by 0
  15. 2023Neural Harmonium: An Interpretable Deep Structure for Nonlinear Dynamic System Identification with Application to Audio Processingcited by 0
  16. 2023An extended asymmetric sigmoid with Perceptron (SIGTRON) for imbalanced linear classificationcited by 0
  17. 2025Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifoldscited by 0
  18. 2026Canonical Regularisation of Wide Feature-Learning Neural Networkscited by 0