# Critical initialization destabilizes higher input derivatives in wide scalar-input networks

arXiv: https://arxiv.org/abs/2609.09244 · HTML: https://mlsanity.com/paper/2609.09244/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 21 references that are in the map.
- First posted on arXiv 8 Sep 2026.

## Builds on (most cited first, 18 of 18)

- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [When and why PINNs fail to train: A neural tangent kernel perspective](https://mlsanity.com/paper/2007.14527/) (2020)
- [ReZero is All You Need: Fast Convergence at Large Depth](https://mlsanity.com/paper/2003.04887/) (2020)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes](https://mlsanity.com/paper/1910.12478/) (2019)
- [On the Impact of the Activation Function on Deep Neural Networks Training](https://mlsanity.com/paper/1902.06853/) (2019)
- [Mean Field Residual Networks: On the Edge of Chaos](https://mlsanity.com/paper/1712.08969/) (2017)
- [PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks](https://mlsanity.com/paper/2402.00326/) (2024)
- [Learning in Sinusoidal Spaces with Physics-Informed Neural Networks](https://mlsanity.com/paper/2109.09338/) (2021)
- [Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping](https://mlsanity.com/paper/2110.01765/) (2021)
- [Stable ResNet](https://mlsanity.com/paper/2010.12859/) (2020)
- [Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchies](https://mlsanity.com/paper/2204.01058/) (2022)
- [Scaling ResNets in the Large-depth Regime](https://mlsanity.com/paper/2206.06929/) (2022)
- [Stabilize Deep ResNet with A Sharp Scaling Factor τ](https://mlsanity.com/paper/1903.07120/) (2019)
- [A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networks](https://mlsanity.com/paper/2412.09752/) (2024)

## Abstract (arXiv)

The edge-of-chaos condition preserves first-order input perturbations in wide randomly initialized networks, but physics-informed losses, score matching and derivative regularization depend on higher input derivatives. For smooth scalar-input fully connected networks, using a joint Gaussianity of the finite derivative jet that holds in the infinite-width limit at each fixed depth, we derive mean-field recursions through third order that are exact at the variance fixed point, with finite-depth corrections that decay geometrically. At criticality, the first-derivative variance is depth-invariant, whereas the second-derivative variance grows linearly whenever the activation has nonzero curvature. The resulting third-order system closes on mean-field susceptibilities. For residual networks with branch scale L^{-1/2}, we prove that every fixed finite derivative order has uniformly bounded variance under explicit regularity assumptions. Simulations verify the critical growth laws, the residual bound, and the closed recursion. The results concern initialization, not trained-network performance.