mlsanityData to 7 Oct 2026

On Dissipativity of Cross-Entropy Loss in Training ResNets

arXiv 2405.19013 ↗First posted 29 May 2024cs.LG cs.SY eess.SYcited by 1

Not on the map. Why?

Abstract (arXiv)

The training of ResNets and neural ODEs can be formulated and analyzed from the perspective of optimal control. This paper proposes a dissipative formulation of the training of ResNets and neural ODEs for classification problems by including a variant of the cross-entropy as a regularization in the stage cost. Based on the dissipative formulation of the training, we prove that the trained ResNet exhibit the turnpike phenomenon. We then illustrate that the training exhibits the turnpike phenomenon by training on the two spirals and MNIST datasets. This can be used to find very shallow networks suitable for a given classification task.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 28 of its 48 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

28 of 48 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2018Neural Ordinary Differential Equationscited by 3,733
  6. 2019Deep Equilibrium Modelscited by 603
  7. 2018Large Margin Deep Networks for Classificationcited by 110
  8. 2017Maximum Principle Based Algorithms for Deep Learningcited by 77
  9. 2017Reversible Architectures for Arbitrarily Deep Residual Neural Networkscited by 69
  10. 2021Neural ODE control for classification, approximation and transportcited by 56
  11. 2021Recurrent Equilibrium Networks: Flexible Dynamic Models with Guaranteed Stability and Robustnesscited by 43
  12. 2019Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretationcited by 22
16 more
  1. 2021Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Designcited by 18
  2. 2020Large-time asymptotics in deep learningcited by 14
  3. 2023Unconstrained Parametrization of Dissipative and Contracting Neural Ordinary Differential Equationscited by 8
  4. 2024Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixturescited by 6
  5. 2021Sparsity in long-time control of neural ODEscited by 5
  6. 2014The turnpike property in finite-dimensional nonlinear optimal controlcited by 4
  7. 2015On Turnpike and Dissipativity Properties of Continuous-Time Optimal Control Problemscited by 3
  8. 2019The turnpike property in nonlinear optimal control -- A geometric approachcited by 2
  9. 2019Dissipativity in economic model predictive control: beyond steady-state optimalitycited by 2
  10. 2020Turnpike Properties in Optimal Control: An Overview of Discrete-Time and Continuous-Time Resultscited by 2
  11. 2021On the Turnpike to Design of Deep Neural Nets: Explicit Depth Boundscited by 2
  12. 2020On Continuous-Time Infinite Horizon Optimal Control -- Dissipativity, Stability and Transversalitycited by 1
  13. 2020Linear turnpike theoremcited by 1
  14. 2020Control of port-Hamiltonian systems with minimal energy supplycited by 1
  15. 2021Manifold Turnpikes, Trims and Symmetriescited by 1
  16. 2023Manifold turnpikes of nonlinear port-Hamiltonian descriptor systems under minimal energy supplycited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Towards an Optimal Control Perspective of ResNet Trainingcited by 1