mlsanityData to 7 Oct 2026

Implicit regularization of deep residual networks towards neural ODEs

arXiv 2309.01213 ↗First posted 3 Sep 2023stat.ML cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

Residual neural networks are state-of-the-art deep learning models. Their continuous-depth analog, neural ordinary differential equations (ODEs), are also widely used. Despite their success, the link between the discrete and continuous models still lacks a solid mathematical foundation. In this article, we take a step in this direction by establishing an implicit regularization of deep residual networks towards neural ODEs, for nonlinear networks trained with gradient flow. We prove that if the network is initialized as a discretization of a neural ODE, then such a discretization holds throughout training. Our results are valid for a finite training time, and also as the training time tends to infinity provided that the network satisfies a Polyak-Lojasiewicz condition. Importantly, this condition holds for a family of residual networks where the residuals are two-layer perceptrons with an overparameterization in width that is only linear, and implies the convergence of gradient flow to a global minimum. Numerical experiments illustrate our results.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 43 of its 64 references that are in the map. First posted on arXiv 3 Sep 2023.

Builds on

43 of 64 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  9. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  10. 2019Augmented Neural ODEscited by 396
  11. 2017Stable Architectures for Deep Neural Networkscited by 340
  12. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
31 more
  1. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  2. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  3. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 215
  4. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  5. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  6. 2020The Lipschitz Constant of Self-Attentioncited by 171
  7. 2020Dissecting Neural ODEscited by 150
  8. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  9. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  10. 2023The emergence of clusters in self-attention dynamicscited by 133
  11. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
  12. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  13. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  14. 2018A Mean-Field Optimal Control Formulation of Deep Learningcited by 67
  15. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  16. 2020Superpolynomial Lower Bounds for Learning One-Layer Neural Networks using Gradient Descentcited by 57
  17. 2019Approximation Capabilities of Neural ODEs and Invertible Residual Networkscited by 56
  18. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  19. 2020Universal Approximation Property of Neural Ordinary Differential Equationscited by 32
  20. 2020On the Global Convergence of Training Deep Linear ResNetscited by 29
  21. 2022Do Residual Neural Networks discretize Neural Ordinary Differential Equations?cited by 27
  22. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  23. 2023Generalization bounds for neural ordinary differential equations and deep residual networkscited by 21
  24. 2022Scaling ResNets in the Large-depth Regimecited by 18
  25. 2019Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networkscited by 16
  26. 2019Global Convergence of Gradient Descent for Deep Linear Residual Networkscited by 16
  27. 2021Scaling Properties of Deep Residual Networkscited by 16
  28. 2021Stateful ODE-Nets using Basis Function Expansionscited by 14
  29. 2021Global convergence of ResNets: From finite to infinite width using linear parameterizationcited by 9
  30. 2022Fitting an immersed submanifold to data via Sussmann's orbit theoremcited by 4
  31. 2022Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 7; 2026: 4.

20 papers.

Sort:

By area:

  1. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  2. 2023Gaussian Interpolation Flowscited by 17
  3. 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
  4. 2025The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagramcited by 13
  5. 2024Towards Understanding the Universality of Transformers for Next-Token Predictioncited by 9
  6. 2024Feedback Favors the Generalization of Neural ODEscited by 8
  7. 2024Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transportcited by 7
  8. 2025Transformative or Conservative? Conservation laws for ResNets and Transformerscited by 7
  9. 2026Clustering in Deep Stochastic Transformerscited by 5
  10. 2024Phase-driven Domain Generalizable Learning for Nonstationary Time Seriescited by 4
  11. 2024DFORM: Diffeomorphic vector field alignment for assessing dynamics across learned modelscited by 3
  12. 2024Generalization of Scaled Deep ResNets in the Mean-Field Regimecited by 3
8 more
  1. 2024The Implicit Bias of Structured State Space Models Can Be Poisoned With Clean Labelscited by 3
  2. 2025Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μP Parametrizationcited by 3
  3. 2025Bridging Neural ODE and ResNet: A Formal Error Bound for Safety Verificationcited by 3
  4. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Expertscited by 1
  5. 2026Correlated initialization of deep residual networkscited by 1
  6. 2025Balancing Multi-modal Sensor Learning via Multi-objective Optimizationcited by 0
  7. 2026Universal Approximation Constraints of Narrow ResNets: The Tunnel Effectcited by 0
  8. 2026Why Geometric Continuity Emerges in Deep Neural Networks: Residual Connections and Rotational Symmetry Breakingcited by 0