# On the Turnpike to Design of Deep Neural Nets: Explicit Depth Bounds

arXiv: https://arxiv.org/abs/2101.03000 · HTML: https://mlsanity.com/paper/2101.03000/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 29 references that are in the map.
- First posted on arXiv 8 Jan 2021.

## Builds on (most cited first, 9 of 9)

- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Maximum Principle Based Algorithms for Deep Learning](https://mlsanity.com/paper/1710.09513/) (2017)
- [Large-time asymptotics in deep learning](https://mlsanity.com/paper/2008.02491/) (2020)
- [On Turnpike and Dissipativity Properties of Continuous-Time Optimal Control Problems](https://mlsanity.com/paper/1509.07315/) (2015)
- [Turnpike Properties in Optimal Control: An Overview of Discrete-Time and Continuous-Time Results](https://mlsanity.com/paper/2011.13670/) (2020)
- [Optimal Neumann control for the 1D wave equation: finite horizon, infinite horizon, boundary tracking terms and the turnpike property](https://mlsanity.com/paper/1505.05017/) (2015)
- [Turnpike Properties in Discrete-Time Mixed-Integer Optimal Control](https://mlsanity.com/paper/2002.02049/) (2020)

## Built on by (most cited first, 2 of 2)

- [Sparsity in long-time control of neural ODEs](https://mlsanity.com/paper/2102.13566/) (2021)
- [On Dissipativity of Cross-Entropy Loss in Training ResNets](https://mlsanity.com/paper/2405.19013/) (2024)

## Abstract (arXiv)

It is well-known that the training of Deep Neural Networks (DNN) can be formalized in the language of optimal control. In this context, this paper leverages classical turnpike properties of optimal control problems to attempt a quantifiable answer to the question of how many layers should be considered in a DNN. The underlying assumption is that the number of neurons per layer -- i.e., the width of the DNN -- is kept constant. Pursuing a different route than the classical analysis of approximation properties of sigmoidal functions, we prove explicit bounds on the required depths of DNNs based on asymptotic reachability assumptions and a dissipativity-inducing choice of the regularization terms in the training problem. Numerical results obtained for the two spiral task data set for classification indicate that the proposed estimates can provide non-conservative depth bounds.