mlsanityData to 7 Oct 2026

Regularization in ResNet with Stochastic Depth

arXiv 2106.03091 ↗First posted 6 Jun 2021stat.ML cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Regularization plays a major role in modern deep learning. From classic techniques such as L1,L2 penalties to other noise-based methods such as Dropout, regularization often yields better generalization properties by avoiding overfitting. Recently, Stochastic Depth (SD) has emerged as an alternative regularization technique for residual neural networks (ResNets) and has proven to boost the performance of ResNet on many tasks [Huang et al., 2016]. Despite the recent success of SD, little is known about this technique from a theoretical perspective. This paper provides a hybrid analysis combining perturbation analysis and signal propagation to shed light on different regularization effects of SD. Our analysis allows us to derive principled guidelines for choosing the survival rates used for training with SD.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

26 of 38 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  5. 2016Deep Networks with Stochastic Depthcited by 973
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  8. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  9. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  10. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  11. 2016Deep Information Propagationcited by 194
  12. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
14 more
  1. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
  2. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  3. 2013Dropout Training as Adaptive Regularizationcited by 89
  4. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  5. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
  6. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  7. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  8. 2020Stable ResNetcited by 49
  9. 2020Tensor Programs III: Neural Matrix Lawscited by 47
  10. 2020Explicit Regularisation in Gaussian Noise Injectionscited by 37
  11. 2020Robust Pruning at Initializationcited by 31
  12. 2014On the Inductive Bias of Dropoutcited by 19
  13. 2017Dropout as a Low-Rank Regularizer for Matrix Factorizationcited by 14
  14. 2019On Dropout and Nuclear Norm Regularizationcited by 12

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 1; 2025: 1.

7 papers.

  1. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  2. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  3. 2024KARINA: An Efficient Deep Learning Model for Global Weather Forecastcited by 9
  4. 2023Fast Propagation is Better: Accelerating Single-Step Adversarial Training via Sampling Subnetworkscited by 8
  5. 2025Deep Neural Networks Inspired by Differential Equationscited by 2
  6. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1
  7. 2023Exploring the Robustness of Decentralized Training for Large Language Modelscited by 1