# A Qualitative Test-Risk Mechanism for Scaling Behavior in Normalized Residual Networks

arXiv: https://arxiv.org/abs/2605.08297 · HTML: https://mlsanity.com/paper/2605.08297/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 32 references that are in the map.
- First posted on arXiv 8 May 2026.

## Builds on (most cited first, 12 of 12)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks](https://mlsanity.com/paper/1707.09564/) (2017)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [A Constructive Prediction of the Generalization Error Across Scales](https://mlsanity.com/paper/1909.12673/) (2019)
- [Fixup Initialization: Residual Learning Without Normalization](https://mlsanity.com/paper/1901.09321/) (2019)
- [Learning to Grow Pretrained Models for Efficient Transformer Training](https://mlsanity.com/paper/2303.00980/) (2023)
- [The Depth-to-Width Interplay in Self-Attention](https://mlsanity.com/paper/2006.12467/) (2020)
- [Compression for Better: A General and Stable Lossless Compression Framework](https://mlsanity.com/paper/2412.06868/) (2024)
- [Smaller generalization error derived for a deep residual neural network compared to shallow networks](https://mlsanity.com/paper/2010.01887/) (2020)
- [On the Generalization Behavior of Deep Residual Networks From a Dynamical System Perspective](https://mlsanity.com/paper/2602.20921/) (2026)

## Built on by (most cited first, 4 of 4)

- [Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimension](https://mlsanity.com/paper/2607.24887/) (2026)
- [Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion](https://mlsanity.com/paper/2608.14664/) (2026)
- [Coupled Scaling: A Representational Accessibility Framework for Neural Scaling Laws](https://mlsanity.com/paper/2609.03533/) (2026)
- [Is Human-Readable Text Necessary for Effective LLM Fine-Tuning?](https://mlsanity.com/paper/2609.35868/) (2026)

## Abstract (arXiv)

The scaling behavior, in which test performance often improves as model size and data increase, is a central empirical phenomenon in modern deep learning, yet its theoretical basis remains incomplete. In this paper, we study depth expansion in normalized residual networks: starting from a trained model in an old hypothesis class, we insert a new residual block at an intermediate layer and ask when such an expansion can yield a provable improvement in test risk. We develop a unified framework that decomposes this question into representational gain, optimization gain, and generalization transfer. First, under a first-order descent condition near zero initialization, we prove that the expanded hypothesis class contains an auxiliary jumpboard model with strictly smaller population risk than the original model. Second, under norm control tailored to post-normalized residual architectures, we establish a norm-based Rademacher complexity bound for the expanded model class. These ingredients lead to two complementary test-risk guarantees: one route passes through population risk and is tighter when a positive population margin is available, while the other works directly at the train/test level, avoids Hoeffding transfer, and is more robust in degenerate regimes. Together, these results provide a theorem-driven mechanism under which residual depth expansion can improve test performance in normalized residual networks. More broadly, they suggest that scaling is inherently joint: depth creates new improving directions, width enhances the finite-sample observability of weak signals, and data determines whether the statistical cost of expansion can be controlled.