# Why ResNet Works? Residuals Generalize

arXiv: https://arxiv.org/abs/1904.01367 · HTML: https://mlsanity.com/paper/1904.01367/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2019.

## Built on by (most cited first, 30 of 39)

- [SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning](https://mlsanity.com/paper/2410.09754/) (2024)
- [Exploring Sequence Feature Alignment for Domain Adaptive Detection Transformers](https://mlsanity.com/paper/2107.12636/) (2021)
- [Mean Teacher DETR with Masked Feature Alignment: A Robust Domain Adaptive Detection Transformer Framework](https://mlsanity.com/paper/2310.15646/) (2023)
- [Progressive Feedforward Collapse of ResNet Training](https://mlsanity.com/paper/2405.00985/) (2024)
- [On Measuring Excess Capacity in Neural Networks](https://mlsanity.com/paper/2202.08070/) (2022)
- [Dispersed Pixel Perturbation-based Imperceptible Backdoor Trigger for Image Classifier Models](https://mlsanity.com/paper/2208.09336/) (2022)
- [Norm-based Generalization Bounds for Compositionally Sparse Neural Networks](https://mlsanity.com/paper/2301.12033/) (2023)
- [Towards Signal Processing In Large Language Models](https://mlsanity.com/paper/2406.10254/) (2024)
- [radarODE: An ODE-Embedded Deep Learning Model for Contactless ECG Reconstruction from Millimeter-Wave Radar](https://mlsanity.com/paper/2408.01672/) (2024)
- [How You Start Matters for Generalization](https://mlsanity.com/paper/2206.08558/) (2022)
- [Phase-driven Domain Generalizable Learning for Nonstationary Time Series](https://mlsanity.com/paper/2402.05960/) (2024)
- [Variational Learning of Gaussian Process Latent Variable Models through Stochastic Gradient Annealed Importance Sampling](https://mlsanity.com/paper/2408.06710/) (2024)
- [Finger-NestNet: Interpretable Fingerphoto Verification on Smartphone using Deep Nested Residual Network](https://mlsanity.com/paper/2212.05884/) (2022)
- [Boosting Residual Networks with Group Knowledge](https://mlsanity.com/paper/2308.13772/) (2023)
- [MoNetV2: Enhanced Motion Network for Freehand 3D Ultrasound Reconstruction](https://mlsanity.com/paper/2506.15835/) (2025)
- [From High-SNR Radar Signal to ECG: A Transfer Learning Model with Cardio-Focusing Algorithm for Scenarios with Limited Data](https://mlsanity.com/paper/2506.19358/) (2025)
- [MC-GNNAS-Dock: Multi-criteria GNN-based Algorithm Selection for Molecular Docking](https://mlsanity.com/paper/2509.26377/) (2025)
- [Spatiotemporal Observer Design for Predictive Learning of High-Dimensional Data](https://mlsanity.com/paper/2402.15284/) (2024)
- [Edge Detectors Can Make Deep Convolutional Neural Networks More Robust](https://mlsanity.com/paper/2402.16479/) (2024)
- [Bilateral Sharpness-Aware Minimization for Flatter Minima](https://mlsanity.com/paper/2409.13173/) (2024)
- [TITAN: Query-Token based Domain Adaptive Adversarial Learning](https://mlsanity.com/paper/2506.21484/) (2025)
- [Convergence Analysis of Deep Residual Networks](https://mlsanity.com/paper/2205.06571/) (2022)
- [Characterising the Inductive Biases of Neural Networks on Boolean Data](https://mlsanity.com/paper/2505.24060/) (2025)
- [Weather-Driven Agricultural Decision-Making Using Digital Twins Under Imperfect Conditions](https://mlsanity.com/paper/2508.08326/) (2025)
- [Molecular Embedding-Based Algorithm Selection in Protein-Ligand Docking](https://mlsanity.com/paper/2512.02328/) (2025)
- [PRISM: Parallel Reward Integration with Symmetry for MORL](https://mlsanity.com/paper/2602.18277/) (2026)
- [On the Generalization Behavior of Deep Residual Networks From a Dynamical System Perspective](https://mlsanity.com/paper/2602.20921/) (2026)
- [Generalization by design: Shortcuts to Generalization in Deep Learning](https://mlsanity.com/paper/2107.02253/) (2021)
- [SML:Enhance the Network Smoothness with Skip Meta Logit for CTR Prediction](https://mlsanity.com/paper/2210.10725/) (2022)
- [One-Shot Acoustic Matching Of Audio Signals -- Learning to Hear Music In Any Room/ Concert Hall](https://mlsanity.com/paper/2210.15750/) (2022)

## Abstract (arXiv)

Residual connections significantly boost the performance of deep neural networks. However, there are few theoretical results that address the influence of residuals on the hypothesis complexity and the generalization ability of deep neural networks. This paper studies the influence of residual connections on the hypothesis complexity of the neural network in terms of the covering number of its hypothesis space. We prove that the upper bound of the covering number is the same as chain-like neural networks, if the total numbers of the weight matrices and nonlinearities are fixed, no matter whether they are in the residuals or not. This result demonstrates that residual connections may not increase the hypothesis complexity of the neural network compared with the chain-like counterpart. Based on the upper bound of the covering number, we then obtain an mathcal O(1 / √N) margin-based multi-class generalization bound for ResNet, as an exemplary case of any deep neural network with residual connections. Generalization guarantees for similar state-of-the-art neural network architectures, such as DenseNet and ResNeXt, are straight-forward. From our generalization bound, a practical implementation is summarized: to approach a good generalization ability, we need to use regularization terms to control the magnitude of the norms of weight matrices not to increase too much, which justifies the standard technique of weight decay.