mlsanityData to 7 Oct 2026

Deep learning as optimal control problems: models and numerical methods

arXiv 1904.05657 ↗First posted 11 Apr 2019math.OC cs.LG cs.NAcited by 33

Not on the map. Why?

Abstract (arXiv)

We consider recent work of Haber and Ruthotto 2017 and Chang et al. 2018, where deep learning neural networks have been interpreted as discretisations of an optimal control problem subject to an ordinary differential equation constraint. We review the first order conditions for optimality, and the conditions ensuring optimality after discretisation. This leads to a class of algorithms for solving the discrete optimal control problem which guarantee that the corresponding discrete necessary conditions for optimality are fulfilled. The differential equation setting lends itself to learning additional parameters such as the time discretisation. We explore this extension alongside natural constraints (e.g. time steps lie in a simplex). We compare these deep learning algorithms numerically in terms of induced flow and generalisation ability.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 6; 2023: 5; 2024: 4; 2025: 5; 2026: 5.

33 papers.

Sort:

  1. 2024Data Selection via Optimal Control for Language Modelscited by 38
  2. 2021Bridging the Gap between Spatial and Spectral Domains: A Unified Framework for Graph Neural Networkscited by 18
  3. 2023PottsMGNet: A Mathematical Explanation of Encoder-Decoder Based Neural Networkscited by 17
  4. 2021Novel Deep neural networks for solving Bayesian statistical inversecited by 10
  5. 2021Connections between Numerical Algorithms for PDEs and Neural Networkscited by 10
  6. 2022Turnpike in optimal control of PDEs, ResNets, and beyondcited by 10
  7. 2022TO-FLOW: Efficient Continuous Normalizing Flows with Temporal Optimization adjoint with Moving Speedcited by 8
  8. 2022Neural ODE Control for Trajectory Approximation of Continuity Equationcited by 8
  9. 2021Deep Learning Approximation of Diffeomorphisms via Linear-Control Systemscited by 7
  10. 2023Double-well Net for Image Segmentationcited by 7
  11. 2024Differential Equations for Continuous-Time Deep Learningcited by 7
  12. 2021Sparsity in long-time control of neural ODEscited by 5
21 more
  1. 2021Neural Architecture Search via Bregman Iterationscited by 5
  2. 2023A minimax optimal control approach for robust neural ODEscited by 4
  3. 2024A Mathematical Explanation of UNetcited by 4
  4. 2021Spline parameterization of neural network controls for deep learningcited by 3
  5. 2022An Optimal Time Variable Learning Framework for Deep Neural Networkscited by 3
  6. 2023From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networkscited by 3
  7. 2024Online Control-Informed Learningcited by 3
  8. 2025Neural Predictive Control to Coordinate Discrete- and Continuous-Time Models for Time-Series Analysis with Control-Theoretical Improvementscited by 3
  9. 2022Deep neural networks on diffeomorphism groups for optimal shape reparameterizationcited by 2
  10. 2025Topological derivative approach for deep neural network architecture adaptationcited by 2
  11. 2025A Mathematical Explanation of Transformerscited by 2
  12. 2025Control, Optimal Transport and Neural Differential Equations in Supervised Learningcited by 1
  13. 2021Classification with Runge-Kutta networks and feature space augmentationcited by 0
  14. 2022Multisymplectic Formulation of Deep Learning Using Mean--Field Type Control and Nonlinear Stability of Training Algorithmcited by 0
  15. 2023Time Regularization in Optimal Time Variable Learningcited by 0
  16. 2025Weight-Parameterization in Continuous Time Deep Neural Networks for Surrogate Modelingcited by 0
  17. 2026Layerwise goal-oriented adaptivity for neural ODEs: an optimal control perspectivecited by 0
  18. 2026On the Generalization Behavior of Deep Residual Networks From a Dynamical System Perspectivecited by 0
  19. 2026Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Modelscited by 0
  20. 2026Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability-Observability Testscited by 0
  21. 2026An optimal control approach for neural network architecture adaptation with a posteriori error estimationcited by 0