# Sparsity in long-time control of neural ODEs

arXiv: https://arxiv.org/abs/2102.13566 · HTML: https://mlsanity.com/paper/2102.13566/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 50 references that are in the map.
- First posted on arXiv 26 Feb 2021.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Large Margin Deep Networks for Classification](https://mlsanity.com/paper/1803.05598/) (2018)
- [Optimal Approximation with Sparsely Connected Deep Neural Networks](https://mlsanity.com/paper/1705.01714/) (2017)
- [Maximum Principle Based Algorithms for Deep Learning](https://mlsanity.com/paper/1710.09513/) (2017)
- [Deep Learning via Dynamical Systems: An Approximation Perspective](https://mlsanity.com/paper/1912.10382/) (2019)
- [Understanding Deep Convolutional Networks](https://mlsanity.com/paper/1601.04920/) (2016)
- [Deep learning as optimal control problems: models and numerical methods](https://mlsanity.com/paper/1904.05657/) (2019)
- [Structure preserving deep learning](https://mlsanity.com/paper/2006.03364/) (2020)
- [Large-time asymptotics in deep learning](https://mlsanity.com/paper/2008.02491/) (2020)
- [The turnpike property in finite-dimensional nonlinear optimal control](https://mlsanity.com/paper/1402.3263/) (2014)
- [Turnpike in Lipschitz-nonlinear optimal control](https://mlsanity.com/paper/2011.11091/) (2020)
- [Mean-Field Sparse Optimal Control](https://mlsanity.com/paper/1402.5657/) (2014)
- [On the Turnpike to Design of Deep Neural Nets: Explicit Depth Bounds](https://mlsanity.com/paper/2101.03000/) (2021)
- [Infinite horizon sparse optimal control](https://mlsanity.com/paper/1602.08931/) (2016)
- [Sparse and Switching Infinite Horizon Optimal Control with Mixed-Norm Penalizations](https://mlsanity.com/paper/1808.10659/) (2018)
- [The Finite-Time Turnpike Phenomenon for Optimal Control Problems: Stabilization by Non-Smooth Tracking Terms](https://mlsanity.com/paper/2006.07051/) (2020)

## Built on by (most cited first, 5 of 5)

- [Interplay between depth and width for interpolation in neural ODEs](https://mlsanity.com/paper/2401.09902/) (2024)
- [Cluster-based classification with neural ODEs via control](https://mlsanity.com/paper/2312.13807/) (2023)
- [On Dissipativity of Cross-Entropy Loss in Training ResNets](https://mlsanity.com/paper/2405.19013/) (2024)
- [Towards an Optimal Control Perspective of ResNet Training](https://mlsanity.com/paper/2506.21453/) (2025)
- [Long-Time Trajectory Approximation via SA-NODEs: Model Predictive and Floquet Strategies](https://mlsanity.com/paper/2608.10738/) (2026)

## Abstract (arXiv)

We consider the neural ODE and optimal control perspective of supervised learning, with ℓ1-control penalties, where rather than only minimizing a final cost (the empirical risk) for the state, we integrate this cost over the entire time horizon. We prove that any optimal control (for this cost) vanishes beyond some positive stopping time. When seen in the discrete-time context, this result entails an ordered sparsity pattern for the parameters of the associated residual neural network: ordered in the sense that these parameters are all 0 beyond a certain layer. Furthermore, we provide a polynomial stability estimate for the empirical risk with respect to the time horizon. This can be seen as a turnpike property, for nonsmooth dynamics and functionals with ℓ1-penalties, and without any smallness assumptions on the data, both of which are new in the literature.