mlsanityData to 7 Oct 2026

Stable ResNet

arXiv 2010.12859 ↗First posted 24 Oct 2020cs.LG stat.MLcited by 49

Not on the map. Why?

Abstract (arXiv)

Deep ResNet architectures have achieved state of the art performance on many tasks. While they solve the problem of gradient vanishing, they might suffer from gradient exploding as the depth becomes large (Yang et al. 2017). Moreover, recent results have shown that ResNet might lose expressivity as the depth goes to infinity (Yang et al. 2017, Hayou et al. 2019). To resolve these issues, we introduce a new class of ResNet architectures, called Stable ResNet, that have the property of stabilizing the gradient while ensuring expressivity in the infinite depth limit.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 13; 2024: 7; 2025: 7; 2026: 10.

49 papers.

Sort:

By area:

  1. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  2. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  3. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  4. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  5. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 59
  6. 2025The Curse of Depth in Large Language Modelscited by 58
  7. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  8. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  9. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
  10. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  11. 2023Simplifying Transformer Blockscited by 37
  12. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
37 more
  1. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  2. 2023ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connectioncited by 25
  3. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  4. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  5. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  6. 2021Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networkscited by 20
  7. 2023Leave-one-out Distinguishability in Machine Learningcited by 20
  8. 2022Scaling ResNets in the Large-depth Regimecited by 18
  9. 2023Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Modelscited by 14
  10. 2025The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagramcited by 13
  11. 2025Always Skip Attentioncited by 12
  12. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  13. 2024Falcon2-11B Technical Reportcited by 11
  14. 2021Wide stochastic networks: Gaussian limit and PAC-Bayesian trainingcited by 10
  15. 2024Understanding and Minimising Outlier Features in Neural Network Trainingcited by 10
  16. 2026Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuningcited by 8
  17. 2021Regularization in ResNet with Stochastic Depthcited by 7
  18. 2023Field theory for optimal signal propagation in ResNetscited by 6
  19. 2025The Importance of Being Lazy: Scaling Limits of Continual Learningcited by 6
  20. 2026Clustering in Deep Stochastic Transformerscited by 5
  21. 2023Differential Equation Scaling Limits of Shaped and Unshaped Neural Networkscited by 4
  22. 2025Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Sizecited by 4
  23. 2022A DNN Framework for Learning Lagrangian Drift With Uncertaintycited by 3
  24. 2022An Optimal Time Variable Learning Framework for Deep Neural Networkscited by 3
  25. 2024Generalization of Scaled Deep ResNets in the Mean-Field Regimecited by 3
  26. 2025A Proof of Learning Rate Transfer under μPcited by 3
  27. 2026On the Infinite Width and Depth Limits of Predictive Coding Networkscited by 2
  28. 2026Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depthcited by 2
  29. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1
  30. 2023A note on regularised NTK dynamics with an application to PAC-Bayesian trainingcited by 1
  31. 2025Probability Distribution Collapse: A Critical Bottleneck to Compact Unsupervised Neural Grammar Inductioncited by 0
  32. 2026Optimal Initialization in Depth: Lyapunov Initialization and Limit Theorems for Deep Leaky ReLU Networkscited by 0
  33. 2026Collective Kernel EFT for Pre-activation ResNetscited by 0
  34. 2026Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flowscited by 0
  35. 2026Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Productcited by 0
  36. 2026Critical initialization destabilizes higher input derivatives in wide scalar-input networkscited by 0
  37. 2026Stable initialization without the CLTcited by 0