Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifolds
Not on the map. Why?
Abstract (arXiv)
The high-dimensional parameter space of deep neural networks -- the neuromanifold -- is endowed with a unique metric tensor defined by the Fisher information. Reliable and scalable computation of this metric tensor is valuable for theorists and practitioners. Focusing on neural classifiers, we return to a low-dimensional space of probability distributions, which we call the core space, and examine the spectrum and envelopes of its Fisher information matrix. We extend our discoveries there to deterministic bounds for the metric tensor on the neuromanifold. We introduce an unbiased random estimator based on Hutchinson's trace method and derive related bounds. It can be evaluated efficiently with a single backward pass per batch, with a standard deviation bounded by the true value up to scaling.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 19 May 2025.
Builds on
33 of 59 references.
References · 33
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
- 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
- 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
- 2014New insights and perspectives on the natural gradient methodcited by 435
21 more
- 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
- 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
- 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
- 2013Revisiting Natural Gradient for Deep Networkscited by 157
- 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
- 2020The Hessian Penalty: A Weak Prior for Unsupervised Disentanglementcited by 90
- 2019BackPACK: Packing more into backpropcited by 77
- 2020Hutch++: Optimal Stochastic Trace Estimationcited by 62
- 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 61
- 2019Pathological spectra of the Fisher information metric and its variants in deep neural networkscited by 32
- 2013Riemannian metrics for neural networks I: feedforward networkscited by 29
- 2023Hutchinson Trace Estimation for High-Dimensional and High-Order Physics-Informed Neural Networkscited by 23
- 2021Tractable structured natural gradient descent using local parameterizationscited by 22
- 2021On the Variance of the Fisher Information for Deep Learningcited by 22
- 2022Visualizing high-dimensional loss landscapes with Hessian directionscited by 16
- 2020A Modern Analysis of Hutchinson's Trace Estimatorcited by 11
- 2016Reducing the Model Order of Deep Neural Networks Using Information Theorycited by 10
- 2024Trade-Offs of Diagonal Fisher Information Matrix Estimatorscited by 10
- 2018Scalable Gaussian Process Computations Using Hierarchical Matricescited by 8
- 2019A Geometric Modeling of Occam's Razor in Deep Learningcited by 8
- 2023On Surgical Fine-tuning for Language Encoderscited by 8
Built on by
Not cited yet.