# On the Stability of the Jacobian Matrix in Deep Neural Networks

arXiv: https://arxiv.org/abs/2506.08764 · HTML: https://mlsanity.com/paper/2506.08764/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 45 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Builds on (most cited first, 23 of 23)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [An Introduction to Matrix Concentration Inequalities](https://mlsanity.com/paper/1501.01571/) (2015)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes](https://mlsanity.com/paper/1910.12478/) (2019)
- [Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice](https://mlsanity.com/paper/1711.04735/) (2017)
- [All you need is a good init](https://mlsanity.com/paper/1511.06422/) (2015)
- [On the Impact of the Activation Function on Deep Neural Networks Training](https://mlsanity.com/paper/1902.06853/) (2019)
- [Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win](https://mlsanity.com/paper/2010.03533/) (2020)
- [Products of Many Large Random Matrices and Gradients in Deep Neural Networks](https://mlsanity.com/paper/1812.05994/) (2018)
- [Recent Advances on Neural Network Pruning at Initialization](https://mlsanity.com/paper/2103.06460/) (2021)
- [How does Weight Correlation Affect the Generalisation Ability of Deep Neural Networks](https://mlsanity.com/paper/2010.05983/) (2020)
- [Robust Pruning at Initialization](https://mlsanity.com/paper/2002.08797/) (2020)
- [Data-dependent Initializations of Convolutional Neural Networks](https://mlsanity.com/paper/1511.06856/) (2015)
- [Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of Multilayer Perceptron: The Haar Orthogonal Case](https://mlsanity.com/paper/2103.13466/) (2021)
- [Free Probability for predicting the performance of feed-forward fully connected neural networks](https://mlsanity.com/paper/2111.00841/) (2021)

## Built on by (most cited first, 3 of 3)

- [Mining Useful General Data for Low-Resource Domain Adaptation](https://mlsanity.com/paper/2511.07380/) (2025)
- [Mixture-of-Experts as Soft Clustering: A Dual Jacobian-PCA Spectral Geometry Perspective](https://mlsanity.com/paper/2601.11616/) (2026)
- [Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap](https://mlsanity.com/paper/2605.16349/) (2026)

## Abstract (arXiv)

Deep neural networks are known to suffer from exploding or vanishing gradients as depth increases, a phenomenon closely tied to the spectral behavior of the input-output Jacobian. Prior work has identified critical initialization schemes that ensure Jacobian stability, but these analyses are typically restricted to fully connected networks with i.i.d. weights. In this work, we go significantly beyond these limitations: we establish a general stability theorem for deep neural networks that accommodates sparsity (such as that introduced by pruning) and non-i.i.d., weakly correlated weights (e.g. induced by training). Our results rely on recent advances in random matrix theory, and provide rigorous guarantees for spectral stability in a much broader class of network models. This extends the theoretical foundation for initialization schemes in modern neural networks with structured and dependent randomness.