mlsanityData to 14 Oct 2026

Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion

arXiv 2608.14664 ↗First posted 1 Aug 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

How can we determine whether a trained neural network is already deep enough? We study this under a fixed function-preserving residual-growth protocol specifying insertion locations, residual families, zero-output initializations, and zero-state first-order updates. We define first-order residual depth saturation as the absence of a strict local decrease from every admissible insertion. We prove residual non-degeneracy is necessary and sufficient: additional depth has first-order value exactly when conditional activation gradients have a nonzero projection onto at least one admissible residual tangent space. This boundary is shared by descent-compatible zero-state updates and invariant under regular local reparameterizations preserving that tangent space. Under residual-signal realizability, raw activation-gradient vanishing exactly certifies saturation. Across ResNets, GPT-2-style models, and continued-pretrained Pythia checkpoints, the maximum activation-gradient norm decreases toward a low-signal regime with depth. Function-preserving growth also achieves converged performance competitive with training from scratch. These results support activation-gradient magnitude as a conservative diagnostic of the remaining empirical first-order value of residual depth.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 1 Aug 2026.

Builds on

21 of 33 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2016Deep Networks with Stochastic Depthcited by 979
  5. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  6. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
  7. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  8. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  9. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 83
  10. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 66
  11. 2016Network Morphismcited by 65
  12. 2019Splitting Steepest Descent for Growing Neural Architecturescited by 55
9 more
  1. 2020On the Transformer Growth for Progressive BERT Trainingcited by 49
  2. 2021Firefly Neural Architecture Descent: a General Approach for Growing Neural Networkscited by 49
  3. 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Trainingcited by 43
  4. 2020The Depth-to-Width Interplay in Self-Attentioncited by 42
  5. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
  6. 2019AutoGrow: Automatic Layer Growing in Deep Convolutional Networkscited by 26
  7. 2023Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptationcited by 8
  8. 2026A Qualitative Test-Risk Mechanism for Scaling Behavior in Normalized Residual Networkscited by 4
  9. 2025Topological derivative approach for deep neural network architecture adaptationcited by 2

Built on by

Not cited yet.