Critical initialization destabilizes higher input derivatives in wide scalar-input networks
Not on the map. Why?
Abstract (arXiv)
The edge-of-chaos condition preserves first-order input perturbations in wide randomly initialized networks, but physics-informed losses, score matching and derivative regularization depend on higher input derivatives. For smooth scalar-input fully connected networks, using a joint Gaussianity of the finite derivative jet that holds in the infinite-width limit at each fixed depth, we derive mean-field recursions through third order that are exact at the variance fixed point, with finite-depth corrections that decay geometrically. At criticality, the first-derivative variance is depth-invariant, whereas the second-derivative variance grows linearly whenever the activation has nonzero curvature. The resulting third-order system closes on mean-field susceptibilities. For residual networks with branch scale L^{-1/2}, we prove that every fixed finite derivative order has uniformly bounded variance under explicit regularity assumptions. Simulations verify the critical growth laws, the residual bound, and the closed recursion. The results concern initialization, not trained-network performance.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 21 references that are in the map. First posted on arXiv 8 Sep 2026.
Builds on
18 of 21 references.
References · 18
- 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
- 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 629
- 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
- 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
- 2016Deep Information Propagationcited by 194
- 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
- 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
- 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
- 2024PirateNets: Physics-informed Deep Learning with Residual Adaptive Networkscited by 70
- 2021Learning in Sinusoidal Spaces with Physics-Informed Neural Networkscited by 61
6 more
- 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
- 2020Stable ResNetcited by 49
- 2022Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchiescited by 24
- 2022Scaling ResNets in the Large-depth Regimecited by 18
- 2019Stabilize Deep ResNet with A Sharp Scaling Factor τcited by 10
- 2024A Quasilinear Algorithm for Computing Higher-Order Derivatives of Deep Feed-Forward Neural Networkscited by 2
Built on by
Not cited yet.