# Fisher-Rao Metric, Geometry, and Complexity of Neural Networks

arXiv: https://arxiv.org/abs/1711.01530 · HTML: https://mlsanity.com/paper/1711.01530/

## Key facts

- Cited by 100 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2017.

## Built on by (most cited first, 30 of 100)

- [ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks](https://mlsanity.com/paper/2102.11600/) (2021)
- [Surrogate Gap Minimization Improves Sharpness-Aware Training](https://mlsanity.com/paper/2203.08065/) (2022)
- [Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation](https://mlsanity.com/paper/2211.11004/) (2022)
- [Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks](https://mlsanity.com/paper/2103.00476/) (2021)
- [Sharpness-Aware Training for Free](https://mlsanity.com/paper/2205.14083/) (2022)
- [A Modern Look at the Relationship between Sharpness and Generalization](https://mlsanity.com/paper/2302.07011/) (2023)
- [Model Complexity of Deep Learning: A Survey](https://mlsanity.com/paper/2103.05127/) (2021)
- [The alignment property of SGD noise and how it helps select flat minima: A stability analysis](https://mlsanity.com/paper/2207.02628/) (2022)
- [ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients](https://mlsanity.com/paper/2301.11300/) (2023)
- [The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descent](https://mlsanity.com/paper/2305.17490/) (2023)
- [Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape](https://mlsanity.com/paper/2201.08025/) (2022)
- [PAC-Bayes Information Bottleneck](https://mlsanity.com/paper/2109.14509/) (2021)
- [Randomized Adversarial Training via Taylor Expansion](https://mlsanity.com/paper/2303.10653/) (2023)
- [Exploring Temporal Information Dynamics in Spiking Neural Networks](https://mlsanity.com/paper/2211.14406/) (2022)
- [Fantastic Generalization Measures are Nowhere to be Found](https://mlsanity.com/paper/2309.13658/) (2023)
- [Bounding the Rademacher Complexity of Fourier neural operators](https://mlsanity.com/paper/2209.05150/) (2022)
- [The Geometry of Neural Nets' Parameter Spaces Under Reparametrization](https://mlsanity.com/paper/2302.07384/) (2023)
- [Effective dimension of machine learning models](https://mlsanity.com/paper/2112.04807/) (2021)
- [Generalization Error Bounds for Deep Neural Networks Trained by SGD](https://mlsanity.com/paper/2206.03299/) (2022)
- [A Universal Class of Sharpness-Aware Minimization Algorithms](https://mlsanity.com/paper/2406.03682/) (2024)
- [Using Degeneracy in the Loss Landscape for Mechanistic Interpretability](https://mlsanity.com/paper/2405.10927/) (2024)
- [Effects of Data Geometry in Early Deep Learning](https://mlsanity.com/paper/2301.00008/) (2022)
- [On the Convergence of Shallow Neural Network Training with Randomly Masked Neurons](https://mlsanity.com/paper/2112.02668/) (2021)
- [FIT: A Metric for Model Sensitivity](https://mlsanity.com/paper/2210.08502/) (2022)
- [A numerical approximation method for the Fisher-Rao distance between multivariate normal distributions](https://mlsanity.com/paper/2302.08175/) (2023)
- [Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning](https://mlsanity.com/paper/2310.08782/) (2023)
- [Generalization of Quantum Machine Learning Models Using Quantum Fisher Information Metric](https://mlsanity.com/paper/2303.13462/) (2023)
- [CR-SAM: Curvature Regularized Sharpness-Aware Minimization](https://mlsanity.com/paper/2312.13555/) (2023)
- [Information-Theoretic Generalization Bounds for Deep Neural Networks](https://mlsanity.com/paper/2404.03176/) (2024)
- [Predicting the generalization gap in neural networks using topological data analysis](https://mlsanity.com/paper/2203.12330/) (2022)

## Abstract (arXiv)

We study the relationship between geometry and capacity measures for deep neural networks from an invariance viewpoint. We introduce a new notion of capacity --- the Fisher-Rao norm --- that possesses desirable invariance properties and is motivated by Information Geometry. We discover an analytical characterization of the new capacity measure, through which we establish norm-comparison inequalities and further show that the new measure serves as an umbrella for several existing norm-based complexity measures. We discuss upper bounds on the generalization error induced by the proposed measure. Extensive numerical experiments on CIFAR-10 support our theoretical findings. Our theoretical analysis rests on a key structural lemma about partial derivatives of multi-layer rectifier networks.