# Maximum Principle Based Algorithms for Deep Learning

arXiv: https://arxiv.org/abs/1710.09513 · HTML: https://mlsanity.com/paper/1710.09513/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2017.

## Built on by (most cited first, 30 of 78)

- [Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models](https://mlsanity.com/paper/2203.06904/) (2022)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [Closed-form Continuous-time Neural Models](https://mlsanity.com/paper/2106.13898/) (2021)
- [Data Selection via Optimal Control for Language Models](https://mlsanity.com/paper/2410.07064/) (2024)
- [The Mean-Field Dynamics of Transformers](https://mlsanity.com/paper/2512.01868/) (2025)
- [Neural Delay Differential Equations](https://mlsanity.com/paper/2102.10801/) (2021)
- [Training Free Guided Flow Matching with Optimal Control](https://mlsanity.com/paper/2410.18070/) (2024)
- [Towards Robust Neural Networks via Close-loop Control](https://mlsanity.com/paper/2102.01862/) (2021)
- [On Numerical Integration in Neural Ordinary Differential Equations](https://mlsanity.com/paper/2206.07335/) (2022)
- [On Robust Prefix-Tuning for Text Classification](https://mlsanity.com/paper/2203.10378/) (2022)
- [Second-Order Neural ODE Optimizer](https://mlsanity.com/paper/2109.14158/) (2021)
- [Turnpike in optimal control of PDEs, ResNets, and beyond](https://mlsanity.com/paper/2202.04097/) (2022)
- [An Optimal Control Framework for Joint-channel Parallel MRI Reconstruction without Coil Sensitivities](https://mlsanity.com/paper/2109.09738/) (2021)
- [Coupling parameter and particle dynamics for adaptive sampling in Neural Galerkin schemes](https://mlsanity.com/paper/2306.15630/) (2023)
- [Personalized Algorithm Generation: A Case Study in Learning ODE Integrators](https://mlsanity.com/paper/2105.01303/) (2021)
- [Neural ODE Control for Trajectory Approximation of Continuity Equation](https://mlsanity.com/paper/2205.09241/) (2022)
- [Deep Neural Network Approximation of Invariant Functions through Dynamical Systems](https://mlsanity.com/paper/2208.08707/) (2022)
- [On the regularized risk of distributionally robust learning over deep neural networks](https://mlsanity.com/paper/2109.06294/) (2021)
- [Deep Learning Approximation of Diffeomorphisms via Linear-Control Systems](https://mlsanity.com/paper/2110.12393/) (2021)
- [TO-FLOW: Efficient Continuous Normalizing Flows with Temporal Optimization adjoint with Moving Speed](https://mlsanity.com/paper/2203.10335/) (2022)
- [Linear convergence of a policy gradient method for some finite horizon continuous time control problems](https://mlsanity.com/paper/2203.11758/) (2022)
- [On the Universal Approximation Property of Deep Fully Convolutional Neural Networks](https://mlsanity.com/paper/2211.14047/) (2022)
- [Differential Equations for Continuous-Time Deep Learning](https://mlsanity.com/paper/2401.03965/) (2024)
- [Adjoint Matching through the Lens of the Stochastic Maximum Principle in Optimal Control](https://mlsanity.com/paper/2604.08580/) (2026)
- [Sparsity in long-time control of neural ODEs](https://mlsanity.com/paper/2102.13566/) (2021)
- [Diffusion Mechanism in Residual Neural Network: Theory and Applications](https://mlsanity.com/paper/2105.03155/) (2021)
- [Self-Healing Robust Neural Networks via Closed-Loop Control](https://mlsanity.com/paper/2206.12963/) (2022)
- [Approximation capabilities of measure-preserving neural networks](https://mlsanity.com/paper/2106.10911/) (2021)
- [Neural Piecewise-Constant Delay Differential Equations](https://mlsanity.com/paper/2201.00960/) (2022)
- [ModLaNets: Learning Generalisable Dynamics via Modularity and Physical Inductive Bias](https://mlsanity.com/paper/2206.12325/) (2022)

## Abstract (arXiv)

The continuous dynamical system approach to deep learning is explored in order to devise alternative frameworks for training algorithms. Training is recast as a control problem and this allows us to formulate necessary optimality conditions in continuous time using the Pontryagin's maximum principle (PMP). A modification of the method of successive approximations is then used to solve the PMP, giving rise to an alternative training algorithm for deep learning. This approach has the advantage that rigorous error estimates and convergence results can be established. We also show that it may avoid some pitfalls of gradient-based methods, such as slow convergence on flat landscapes near saddle points. Furthermore, we demonstrate that it obtains favorable initial convergence rate per-iteration, provided Hamiltonian maximization can be efficiently carried out - a step which is still in need of improvement. Overall, the approach opens up new avenues to attack problems associated with deep learning, such as trapping in slow manifolds and inapplicability of gradient-based methods for discrete trainable variables.