# Deep learning as optimal control problems: models and numerical methods

arXiv: https://arxiv.org/abs/1904.05657 · HTML: https://mlsanity.com/paper/1904.05657/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2019.

## Built on by (most cited first, 30 of 34)

- [Data Selection via Optimal Control for Language Models](https://mlsanity.com/paper/2410.07064/) (2024)
- [Bridging the Gap between Spatial and Spectral Domains: A Unified Framework for Graph Neural Networks](https://mlsanity.com/paper/2107.10234/) (2021)
- [PottsMGNet: A Mathematical Explanation of Encoder-Decoder Based Neural Networks](https://mlsanity.com/paper/2307.09039/) (2023)
- [Turnpike in optimal control of PDEs, ResNets, and beyond](https://mlsanity.com/paper/2202.04097/) (2022)
- [Novel Deep neural networks for solving Bayesian statistical inverse](https://mlsanity.com/paper/2102.03974/) (2021)
- [Connections between Numerical Algorithms for PDEs and Neural Networks](https://mlsanity.com/paper/2107.14742/) (2021)
- [Neural ODE Control for Trajectory Approximation of Continuity Equation](https://mlsanity.com/paper/2205.09241/) (2022)
- [Deep Learning Approximation of Diffeomorphisms via Linear-Control Systems](https://mlsanity.com/paper/2110.12393/) (2021)
- [TO-FLOW: Efficient Continuous Normalizing Flows with Temporal Optimization adjoint with Moving Speed](https://mlsanity.com/paper/2203.10335/) (2022)
- [Double-well Net for Image Segmentation](https://mlsanity.com/paper/2401.00456/) (2023)
- [Differential Equations for Continuous-Time Deep Learning](https://mlsanity.com/paper/2401.03965/) (2024)
- [Sparsity in long-time control of neural ODEs](https://mlsanity.com/paper/2102.13566/) (2021)
- [Neural Architecture Search via Bregman Iterations](https://mlsanity.com/paper/2106.02479/) (2021)
- [A minimax optimal control approach for robust neural ODEs](https://mlsanity.com/paper/2310.17584/) (2023)
- [Online Control-Informed Learning](https://mlsanity.com/paper/2410.03924/) (2024)
- [A Mathematical Explanation of UNet](https://mlsanity.com/paper/2410.04434/) (2024)
- [Spline parameterization of neural network controls for deep learning](https://mlsanity.com/paper/2103.00301/) (2021)
- [An Optimal Time Variable Learning Framework for Deep Neural Networks](https://mlsanity.com/paper/2204.08528/) (2022)
- [From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networks](https://mlsanity.com/paper/2307.02279/) (2023)
- [Neural Predictive Control to Coordinate Discrete- and Continuous-Time Models for Time-Series Analysis with Control-Theoretical Improvements](https://mlsanity.com/paper/2508.01833/) (2025)
- [Deep neural networks on diffeomorphism groups for optimal shape reparameterization](https://mlsanity.com/paper/2207.11141/) (2022)
- [Topological derivative approach for deep neural network architecture adaptation](https://mlsanity.com/paper/2502.06885/) (2025)
- [A Mathematical Explanation of Transformers](https://mlsanity.com/paper/2510.03989/) (2025)
- [Control, Optimal Transport and Neural Differential Equations in Supervised Learning](https://mlsanity.com/paper/2503.15105/) (2025)
- [On the Generalization Behavior of Deep Residual Networks From a Dynamical System Perspective](https://mlsanity.com/paper/2602.20921/) (2026)
- [Classification with Runge-Kutta networks and feature space augmentation](https://mlsanity.com/paper/2104.02369/) (2021)
- [Multisymplectic Formulation of Deep Learning Using Mean--Field Type Control and Nonlinear Stability of Training Algorithm](https://mlsanity.com/paper/2207.12242/) (2022)
- [Time Regularization in Optimal Time Variable Learning](https://mlsanity.com/paper/2306.16111/) (2023)
- [Weight-Parameterization in Continuous Time Deep Neural Networks for Surrogate Modeling](https://mlsanity.com/paper/2507.22045/) (2025)
- [Layerwise goal-oriented adaptivity for neural ODEs: an optimal control perspective](https://mlsanity.com/paper/2601.07397/) (2026)

## Abstract (arXiv)

We consider recent work of Haber and Ruthotto 2017 and Chang et al. 2018, where deep learning neural networks have been interpreted as discretisations of an optimal control problem subject to an ordinary differential equation constraint. We review the first order conditions for optimality, and the conditions ensuring optimality after discretisation. This leads to a class of algorithms for solving the discrete optimal control problem which guarantee that the corresponding discrete necessary conditions for optimality are fulfilled. The differential equation setting lends itself to learning additional parameters such as the time discretisation. We explore this extension alongside natural constraints (e.g. time steps lie in a simplex). We compare these deep learning algorithms numerically in terms of induced flow and generalisation ability.