mlsanityData to 7 Oct 2026

Fisher-Rao Metric, Geometry, and Complexity of Neural Networks

arXiv 1711.01530 ↗First posted 5 Nov 2017cs.LG cs.AI stat.MLcited by 99

Not on the map. Why?

Abstract (arXiv)

We study the relationship between geometry and capacity measures for deep neural networks from an invariance viewpoint. We introduce a new notion of capacity --- the Fisher-Rao norm --- that possesses desirable invariance properties and is motivated by Information Geometry. We discover an analytical characterization of the new capacity measure, through which we establish norm-comparison inequalities and further show that the new measure serves as an umbrella for several existing norm-based complexity measures. We discuss upper bounds on the generalization error induced by the proposed measure. Extensive numerical experiments on CIFAR-10 support our theoretical findings. Our theoretical analysis rests on a key structural lemma about partial derivatives of multi-layer rectifier networks.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 21; 2023: 19; 2024: 17; 2025: 10; 2026: 15.

99 papers.

Sort:

By area:

  1. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  2. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  3. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  4. 2021Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networkscited by 134
  5. 2022Sharpness-Aware Training for Freecited by 95
  6. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  7. 2021Model Complexity of Deep Learning: A Surveycited by 64
  8. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  9. 2023ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradientscited by 57
  10. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  11. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  12. 2021PAC-Bayes Information Bottleneckcited by 28
38 more
  1. 2023Randomized Adversarial Training via Taylor Expansioncited by 28
  2. 2022Exploring Temporal Information Dynamics in Spiking Neural Networkscited by 23
  3. 2023Fantastic Generalization Measures are Nowhere to be Foundcited by 19
  4. 2022Bounding the Rademacher Complexity of Fourier neural operatorscited by 16
  5. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  6. 2021Effective dimension of machine learning modelscited by 14
  7. 2022Generalization Error Bounds for Deep Neural Networks Trained by SGDcited by 14
  8. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  9. 2022Effects of Data Geometry in Early Deep Learningcited by 12
  10. 2024Using Degeneracy in the Loss Landscape for Mechanistic Interpretabilitycited by 12
  11. 2021On the Convergence of Shallow Neural Network Training with Randomly Masked Neuronscited by 11
  12. 2023A numerical approximation method for the Fisher-Rao distance between multivariate normal distributionscited by 11
  13. 2023Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learningcited by 11
  14. 2022FIT: A Metric for Model Sensitivitycited by 10
  15. 2023Generalization of Quantum Machine Learning Models Using Quantum Fisher Information Metriccited by 10
  16. 2023CR-SAM: Curvature Regularized Sharpness-Aware Minimizationcited by 10
  17. 2024Information-Theoretic Generalization Bounds for Deep Neural Networkscited by 10
  18. 2022Predicting the generalization gap in neural networks using topological data analysiscited by 9
  19. 2022Lower and Upper Bounds for Numbers of Linear Regions of Graph Convolutional Networkscited by 9
  20. 2022Task Discovery: Finding the Tasks that Neural Networks Generalize oncited by 9
  21. 2022Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernelcited by 7
  22. 2024Boosting Adversarial Training via Fisher-Rao Norm-based Regularizationcited by 7
  23. 2024QGen: On the Ability to Generalize in Quantization Aware Trainingcited by 7
  24. 2021Lower Bounds on the Generalization Error of Nonlinear Learning Modelscited by 6
  25. 2022Weight Expansion: A New Perspective on Dropout and Generalizationcited by 6
  26. 2023Generalization and Estimation Error Bounds for Model-based Neural Networkscited by 6
  27. 2023The Pursuit of Human Labeling: A New Perspective on Unsupervised Learningcited by 6
  28. 2024What Does Softmax Probability Tell Us about Classifiers Ranking Across Diverse Test Conditions?cited by 6
  29. 2024Topological obstruction to the training of shallow ReLU neural networkscited by 6
  30. 2026Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularitiescited by 6
  31. 2023Efficient Activation Function Optimization through Surrogate Modelingcited by 5
  32. 2026Latent Semantic Manifolds in Large Language Modelscited by 5
  33. 2021Minimum sharpness: Scale-invariant parameter-robustness of neural networkscited by 4
  34. 2022Predicting Out-of-Domain Generalization with Neighborhood Invariancecited by 4
  35. 2023Learning Capacity: A Measure of the Effective Dimensionality of a Modelcited by 4
  36. 2023Quantifying Overfitting: Evaluating Neural Network Performance through Analysis of Null Spacecited by 4
  37. 2024A Universal Metric of Dataset Similarity for Cross-silo Federated Learningcited by 4
  38. 2024The Uncanny Valley: Exploring Adversarial Robustness from a Flatness Perspectivecited by 4

50 most cited of 99 All 99 (JSON)