mlsanityData to 7 Oct 2026

Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures

arXiv 2604.11639 ↗First posted 13 Apr 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Modern automatic differentiation frameworks (JAX, PyTorch) return the Hessian of the loss function as a monolithic tensor, without exposing the internal structure of inter-layer interactions. This paper presents an analytical formalism that explicitly decomposes the full Hessian into blocks indexed by the DAG of an arbitrary architecture. The canonical decomposition H = HGN + HT separates the Gauss--Newton component (convex part) from the tensor component (residual curvature responsible for saddle points). For piecewise-linear activations (ReLU), the tensor component of the input Hessian vanishes (HTv,w!equiv!0 a.e., Hfv,w!=!HGNv,w!succeq!0); the full parametric Hessian contains residual terms that do not reduce to the GGN. Building on this decomposition, we introduce diagnostic metrics (inter-layer resonance~R, geometric coupling~C, stable rank~D, GN-Gap) that are estimated stochastically in O(P) time and reveal structural curvature interactions between layers. The theoretical analysis explains exponential decay of resonance in vanilla networks and its preservation under skip connections; empirical validation spans fully connected MLPs (Exp.,1--5) and convolutional architectures (ResNet-18, ~11M~parameters, Exp.,6). When the architecture reduces to a single node, all definitions collapse to the standard Hessian nabla2θL(θ)∈ℝp× p.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 46 references that are in the map. First posted on arXiv 13 Apr 2026.

Builds on

27 of 46 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  5. 2014New insights and perspectives on the natural gradient methodcited by 435
  6. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  7. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  8. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  9. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  10. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  11. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  12. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
15 more
  1. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  2. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  3. 2016Deep Information Propagationcited by 194
  4. 2019Complexity of Linear Regions in Deep Networkscited by 166
  5. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  6. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 156
  7. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 143
  8. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 139
  9. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  10. 2019BackPACK: Packing more into backpropcited by 77
  11. 2019Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learningcited by 71
  12. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  13. 2025The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newtoncited by 17
  14. 2021ViViT: Curvature access through the generalized Gauss-Newton's low-rank structurecited by 16
  15. 2025Local properties of neural networks through the lens of layer-wise Hessianscited by 2

Built on by

Not cited yet.