mlsanityData to 7 Oct 2026

The Hessian perspective into the Nature of Convolutional Neural Networks

arXiv 2305.09088 ↗First posted 16 May 2023cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

While Convolutional Neural Networks (CNNs) have long been investigated and applied, as well as theorized, we aim to provide a slightly different perspective into their nature -- through the perspective of their Hessian maps. The reason is that the loss Hessian captures the pairwise interaction of parameters and therefore forms a natural ground to probe how the architectural aspects of CNN get manifested in its structure and properties. We develop a framework relying on Toeplitz representation of CNNs, and then utilize it to reveal the Hessian structure and, in particular, its rank. We prove tight upper bounds (with linear activations), which closely follow the empirical trend of the Hessian rank and hold in practice in more general settings. Overall, our work generalizes and establishes the key insight that, even in CNNs, the Hessian rank grows as the square root of the number of parameters.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 30 of its 62 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

30 of 62 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  8. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  9. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  10. 2017Implicit Regularization in Matrix Factorizationcited by 318
  11. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  12. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
18 more
  1. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  2. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  3. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  4. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  5. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
  6. 20191D Convolutional Neural Networks and Applications: A Surveycited by 152
  7. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  8. 2018The Singular Values of Convolutional Layerscited by 95
  9. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  10. 2016Inductive Bias of Deep Convolutional Networks through Pooling Geometrycited by 53
  11. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  12. 2020Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networkscited by 45
  13. 2020Theory of Deep Convolutional Neural Networks II: Spherical Analysiscited by 22
  14. 2021Theory of Deep Convolutional Neural Networks III: Approximating Radial Functionscited by 22
  15. 2019Fast-rate PAC-Bayes Generalization Bounds via Shifted Rademacher Processescited by 20
  16. 2019Information-Theoretic Local Minima Characterization and Regularizationcited by 14
  17. 2017Theoretical Properties for Neural Networks with Weight Matrices of Low Displacement Rankcited by 11
  18. 2022Phenomenology of Double Descent in Finite-Width Neural Networkscited by 11

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 5; 2025: 3.

10 papers.

  1. 2024Neglected Hessian component explains mysteries in Sharpness regularizationcited by 12
  2. 2024What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysiscited by 9
  3. 2024Theoretical characterisation of the Gauss-Newton conditioning in Neural Networkscited by 5
  4. 2023A PAC-Bayesian Perspective on the Interpolating Information Criterioncited by 4
  5. 2024Spectral Norm of Convolutional Layers with Circular and Zero Paddingscited by 4
  6. 2025Logits are All We Need to Adapt Closed Modelscited by 4
  7. 2025Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functionscited by 2
  8. 2024Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapescited by 1
  9. 2025Accelerating Neural Network Training Along Sharp and Flat Directionscited by 1
  10. 2023Towards guarantees for parameter isolation in continual learningcited by 0