mlsanityData to 7 Oct 2026

Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach

arXiv 1806.01316 ↗First posted 4 Jun 2018stat.ML cond-mat.dis-nn cs.LGcited by 89

Not on the map. Why?

Abstract (arXiv)

The Fisher information matrix (FIM) is a fundamental quantity to represent the characteristics of a stochastic model, including deep neural networks (DNNs). The present study reveals novel statistics of FIM that are universal among a wide class of DNNs. To this end, we use random weights and large width limits, which enables us to utilize mean field theories. We investigate the asymptotic statistics of the FIM's eigenvalues and reveal that most of them are close to zero while the maximum eigenvalue takes a huge value. Because the landscape of the parameter space is defined by the FIM, it is locally flat in most dimensions, but strongly distorted in others. Moreover, we demonstrate the potential usage of the derived statistics in learning strategies. First, small eigenvalues that induce flatness can be connected to a norm-based capacity measure of generalization ability. Second, the maximum eigenvalue that induces the distortion enables us to quantitatively estimate an appropriately sized learning rate for gradient methods to converge.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 14; 2023: 13; 2024: 10; 2025: 23; 2026: 18.

89 papers.

Sort:

By area:

  1. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  2. 2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
  3. 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
  4. 2023Symmetry Induces Structure and Constraint of Learningcited by 25
  5. 2022Exploring Temporal Information Dynamics in Spiking Neural Networkscited by 23
  6. 2021On the Variance of the Fisher Information for Deep Learningcited by 22
  7. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  8. 2022Foundation Transformerscited by 19
  9. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  10. 2023Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Trainingcited by 19
  11. 2021Non-asymptotic approximations of neural networks by Gaussian processescited by 18
  12. 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
38 more
  1. 2021Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry IIcited by 16
  2. 2022Visualizing high-dimensional loss landscapes with Hessian directionscited by 16
  3. 2022Policy Gradients using Variational Quantum Circuitscited by 15
  4. 2025NoT: Federated Unlearning via Weight Negationcited by 15
  5. 2022Hyperbolic VAE via Latent Gaussian Distributionscited by 14
  6. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 14
  7. 2021Does the Data Induce Capacity Control in Deep Learning?cited by 13
  8. 2025Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulatorcited by 13
  9. 2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
  10. 2022FIT: A Metric for Model Sensitivitycited by 10
  11. 2024Trade-Offs of Diagonal Fisher Information Matrix Estimatorscited by 10
  12. 2024Evolutionary Optimization for Designing Variational Quantum Circuits with High Model Capacitycited by 9
  13. 2023AutoTransfer: AutoML with Knowledge Transfer -- An Application to Graph Neural Networkscited by 8
  14. 2021TENGraD: Time-Efficient Natural Gradient Descent with Exact Fisher-Block Inversioncited by 7
  15. 2022Classical versus Quantum: comparing Tensor Network-based Quantum Circuits on LHC datacited by 7
  16. 2025Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuningcited by 7
  17. 2021Lower Bounds on the Generalization Error of Nonlinear Learning Modelscited by 6
  18. 2023Structural Neural Additive Models: Enhanced Interpretable Machine Learningcited by 6
  19. 2024Fisher Information-based Efficient Curriculum Federated Learning with Large Language Modelscited by 6
  20. 2025Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraintcited by 6
  21. 2021Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of Multilayer Perceptron: The Haar Orthogonal Casecited by 5
  22. 2022Adversarial attacks on neural networks through canonical Riemannian foliationscited by 5
  23. 2023Efficient Activation Function Optimization through Surrogate Modelingcited by 5
  24. 2023Identity Curvature Laplace Approximation for Improved Out-of-Distribution Detectioncited by 4
  25. 2024A Differential Geometric View and Explainability of GNN on Evolving Graphscited by 4
  26. 2025Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retrainingcited by 4
  27. 2023Soft ascent-descent as a stable and flexible alternative to floodingcited by 3
  28. 2024A Two-Scale Complexity Measure for Deep Learning Modelscited by 3
  29. 2024Task2Box: Box Embeddings for Modeling Asymmetric Task Relationshipscited by 3
  30. 2025Evidence on the Regularisation Properties of Maximum-Entropy Reinforcement Learningcited by 3
  31. 2025Fisher information flow in artificial neural networkscited by 3
  32. 2021Approximate Spectral Decomposition of Fisher Information Matrix for Simple ReLU Networkscited by 2
  33. 2021A generalization gap estimation for overparameterized models via the Langevin functional variancecited by 2
  34. 2022A Local Geometric Interpretation of Feature Extraction in Deep Feedforward Neural Networkscited by 2
  35. 2023Deep Unsupervised Learning Using Spike-Timing-Dependent Plasticitycited by 2
  36. 2024Towards the Spectral bias Alleviation by Normalizations in Coordinate Networkscited by 2
  37. 2025Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weightscited by 2
  38. 2025Information-Theoretic Framework for Understanding Modern Machine-Learningcited by 2

50 most cited of 89 All 89 (JSON)