# On Dissipativity of Cross-Entropy Loss in Training ResNets

arXiv: https://arxiv.org/abs/2405.19013 · HTML: https://mlsanity.com/paper/2405.19013/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 48 references that are in the map.
- First posted on arXiv 29 May 2024.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Large Margin Deep Networks for Classification](https://mlsanity.com/paper/1803.05598/) (2018)
- [Maximum Principle Based Algorithms for Deep Learning](https://mlsanity.com/paper/1710.09513/) (2017)
- [Reversible Architectures for Arbitrarily Deep Residual Neural Networks](https://mlsanity.com/paper/1709.03698/) (2017)
- [Neural ODE control for classification, approximation and transport](https://mlsanity.com/paper/2104.05278/) (2021)
- [Recurrent Equilibrium Networks: Flexible Dynamic Models with Guaranteed Stability and Robustness](https://mlsanity.com/paper/2104.05942/) (2021)
- [Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretation](https://mlsanity.com/paper/1902.00407/) (2019)
- [Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Design](https://mlsanity.com/paper/2105.13205/) (2021)
- [Large-time asymptotics in deep learning](https://mlsanity.com/paper/2008.02491/) (2020)
- [Unconstrained Parametrization of Dissipative and Contracting Neural Ordinary Differential Equations](https://mlsanity.com/paper/2304.02976/) (2023)
- [Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures](https://mlsanity.com/paper/2402.02697/) (2024)
- [Sparsity in long-time control of neural ODEs](https://mlsanity.com/paper/2102.13566/) (2021)
- [The turnpike property in finite-dimensional nonlinear optimal control](https://mlsanity.com/paper/1402.3263/) (2014)
- [On Turnpike and Dissipativity Properties of Continuous-Time Optimal Control Problems](https://mlsanity.com/paper/1509.07315/) (2015)
- [The turnpike property in nonlinear optimal control -- A geometric approach](https://mlsanity.com/paper/1903.09069/) (2019)
- [Dissipativity in economic model predictive control: beyond steady-state optimality](https://mlsanity.com/paper/1911.09908/) (2019)
- [Turnpike Properties in Optimal Control: An Overview of Discrete-Time and Continuous-Time Results](https://mlsanity.com/paper/2011.13670/) (2020)
- [On the Turnpike to Design of Deep Neural Nets: Explicit Depth Bounds](https://mlsanity.com/paper/2101.03000/) (2021)
- [On Continuous-Time Infinite Horizon Optimal Control -- Dissipativity, Stability and Transversality](https://mlsanity.com/paper/2001.09601/) (2020)
- [Linear turnpike theorem](https://mlsanity.com/paper/2010.13605/) (2020)
- [Control of port-Hamiltonian systems with minimal energy supply](https://mlsanity.com/paper/2011.10296/) (2020)
- [Manifold Turnpikes, Trims and Symmetries](https://mlsanity.com/paper/2104.03039/) (2021)
- [Manifold turnpikes of nonlinear port-Hamiltonian descriptor systems under minimal energy supply](https://mlsanity.com/paper/2301.09094/) (2023)

## Built on by (most cited first, 1 of 1)

- [Towards an Optimal Control Perspective of ResNet Training](https://mlsanity.com/paper/2506.21453/) (2025)

## Abstract (arXiv)

The training of ResNets and neural ODEs can be formulated and analyzed from the perspective of optimal control. This paper proposes a dissipative formulation of the training of ResNets and neural ODEs for classification problems by including a variant of the cross-entropy as a regularization in the stage cost. Based on the dissipative formulation of the training, we prove that the trained ResNet exhibit the turnpike phenomenon. We then illustrate that the training exhibits the turnpike phenomenon by training on the two spirals and MNIST datasets. This can be used to find very shallow networks suitable for a given classification task.