# Pathological spectra of the Fisher information metric and its variants in deep neural networks

arXiv: https://arxiv.org/abs/1910.05992 · HTML: https://mlsanity.com/paper/1910.05992/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2019.

## Built on by (most cited first, 30 of 32)

- [Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization](https://mlsanity.com/paper/2109.02934/) (2021)
- [Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stability](https://mlsanity.com/paper/2207.12678/) (2022)
- [Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization](https://mlsanity.com/paper/2201.12250/) (2022)
- [On the Activation Function Dependence of the Spectral Bias of Neural Networks](https://mlsanity.com/paper/2208.04924/) (2022)
- [Symmetry Induces Structure and Constraint of Learning](https://mlsanity.com/paper/2309.16932/) (2023)
- [Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training](https://mlsanity.com/paper/2312.00359/) (2023)
- [On the Parameterization of Second-Order Optimization Effective Towards the Infinite Width](https://mlsanity.com/paper/2312.12226/) (2023)
- [Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regime](https://mlsanity.com/paper/2206.02927/) (2022)
- [Trade-Offs of Diagonal Fisher Information Matrix Estimators](https://mlsanity.com/paper/2402.05379/) (2024)
- [Model-centric Data Manifold: the Data Through the Eyes of the Model](https://mlsanity.com/paper/2104.13289/) (2021)
- [Spectral Collapse Drives Loss of Plasticity in Deep Continual Learning](https://mlsanity.com/paper/2509.22335/) (2025)
- [Efficient Activation Function Optimization through Surrogate Modeling](https://mlsanity.com/paper/2301.05785/) (2023)
- [TorchNTK: A Library for Calculation of Neural Tangent Kernels of PyTorch Models](https://mlsanity.com/paper/2205.12372/) (2022)
- [Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights](https://mlsanity.com/paper/2502.04975/) (2025)
- [Information-Theoretic Framework for Understanding Modern Machine-Learning](https://mlsanity.com/paper/2506.07661/) (2025)
- [LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution](https://mlsanity.com/paper/2601.21929/) (2026)
- [Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale](https://mlsanity.com/paper/2606.19491/) (2026)
- [Towards the Spectral bias Alleviation by Normalizations in Coordinate Networks](https://mlsanity.com/paper/2407.17834/) (2024)
- [Decoupling Dynamical Richness from Representation Learning: Towards Practical Measurement](https://mlsanity.com/paper/2410.04264/) (2024)
- [Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity](https://mlsanity.com/paper/2606.21158/) (2026)
- [Far from Asymptopia](https://mlsanity.com/paper/2205.03343/) (2022)
- [A Teacher-Student Perspective on the Dynamics of Learning Near the Optimal Point](https://mlsanity.com/paper/2512.15606/) (2025)
- [An Isotropic Approach to Efficient Uncertainty Quantification with Gradient Norms](https://mlsanity.com/paper/2603.29466/) (2026)
- [Error whitening: Why Gauss-Newton outperforms Newton](https://mlsanity.com/paper/2605.11316/) (2026)
- [Deep Learning in Random Neural Fields: Numerical Experiments via Neural Tangent Kernel](https://mlsanity.com/paper/2202.05254/) (2022)
- [Sparse deep neural networks for nonparametric estimation in high-dimensional sparse regression](https://mlsanity.com/paper/2406.18137/) (2024)
- [Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifolds](https://mlsanity.com/paper/2505.13614/) (2025)
- [Fisher Information, Training and Bias in Fourier Regression Models](https://mlsanity.com/paper/2510.06945/) (2025)
- [Inconsistency-Aware Minimization: Improving Generalization with Unlabeled Data](https://mlsanity.com/paper/2605.31324/) (2026)
- [Effective Dimensionality as an Operator Invariant for Physics-Preserving Constraint Adaptation in Physics-Informed Neural Networks](https://mlsanity.com/paper/2606.06171/) (2026)

## Abstract (arXiv)

The Fisher information matrix (FIM) plays an essential role in statistics and machine learning as a Riemannian metric tensor or a component of the Hessian matrix of loss functions. Focusing on the FIM and its variants in deep neural networks (DNNs), we reveal their characteristic scale dependence on the network width, depth and sample size when the network has random weights and is sufficiently wide. This study covers two widely-used FIMs for regression with linear output and for classification with softmax output. Both FIMs asymptotically show pathological eigenvalue spectra in the sense that a small number of eigenvalues become large outliers depending the width or sample size while the others are much smaller. It implies that the local shape of the parameter space or loss landscape is very sharp in a few specific directions while almost flat in the other directions. In particular, the softmax output disperses the outliers and makes a tail of the eigenvalue density spread from the bulk. We also show that pathological spectra appear in other variants of FIMs: one is the neural tangent kernel; another is a metric for the input signal and feature space that arises from feedforward signal propagation. Thus, we provide a unified perspective on the FIM and its variants that will lead to more quantitative understanding of learning in large-scale DNNs.