mlsanityData to 7 Oct 2026

Sparsity in long-time control of neural ODEs

arXiv 2102.13566 ↗First posted 26 Feb 2021cs.LG math.OC stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

We consider the neural ODE and optimal control perspective of supervised learning, with ℓ1-control penalties, where rather than only minimizing a final cost (the empirical risk) for the state, we integrate this cost over the entire time horizon. We prove that any optimal control (for this cost) vanishes beyond some positive stopping time. When seen in the discrete-time context, this result entails an ordered sparsity pattern for the parameters of the associated residual neural network: ordered in the sense that these parameters are all 0 beyond a certain layer. Furthermore, we provide a polynomial stability estimate for the empirical risk with respect to the time horizon. This can be seen as a turnpike property, for nonsmooth dynamics and functionals with ℓ1-penalties, and without any smallness assumptions on the data, both of which are new in the literature.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 22 of its 50 references that are in the map. First posted on arXiv 26 Feb 2021.

Builds on

22 of 50 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Ordinary Differential Equationscited by 3,733
  3. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  4. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 584
  5. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  6. 2019Augmented Neural ODEscited by 396
  7. 2017Stable Architectures for Deep Neural Networkscited by 340
  8. 2018Large Margin Deep Networks for Classificationcited by 110
  9. 2017Optimal Approximation with Sparsely Connected Deep Neural Networkscited by 90
  10. 2019Deep Learning via Dynamical Systems: An Approximation Perspectivecited by 78
  11. 2017Maximum Principle Based Algorithms for Deep Learningcited by 77
  12. 2016Understanding Deep Convolutional Networkscited by 76
10 more
  1. 2019Deep learning as optimal control problems: models and numerical methodscited by 33
  2. 2020Structure preserving deep learningcited by 30
  3. 2020Large-time asymptotics in deep learningcited by 14
  4. 2014The turnpike property in finite-dimensional nonlinear optimal controlcited by 4
  5. 2020Turnpike in Lipschitz-nonlinear optimal controlcited by 3
  6. 2014Mean-Field Sparse Optimal Controlcited by 2
  7. 2021On the Turnpike to Design of Deep Neural Nets: Explicit Depth Boundscited by 2
  8. 2016Infinite horizon sparse optimal controlcited by 1
  9. 2018Sparse and Switching Infinite Horizon Optimal Control with Mixed-Norm Penalizationscited by 1
  10. 2020The Finite-Time Turnpike Phenomenon for Optimal Control Problems: Stabilization by Non-Smooth Tracking Termscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 1; 2026: 1.

5 papers.

  1. 2024Interplay between depth and width for interpolation in neural ODEscited by 8
  2. 2023Cluster-based classification with neural ODEs via controlcited by 4
  3. 2024On Dissipativity of Cross-Entropy Loss in Training ResNetscited by 1
  4. 2025Towards an Optimal Control Perspective of ResNet Trainingcited by 1
  5. 2026Long-Time Trajectory Approximation via SA-NODEs: Model Predictive and Floquet Strategiescited by 0