# Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective

arXiv: https://arxiv.org/abs/1908.10920 · HTML: https://mlsanity.com/paper/1908.10920/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2019.

## Built on by (most cited first, 27 of 27)

- [On Tilted Losses in Machine Learning: Theory and Applications](https://mlsanity.com/paper/2109.06141/) (2021)
- [Data Selection via Optimal Control for Language Models](https://mlsanity.com/paper/2410.07064/) (2024)
- [From Continuous Dynamics to Graph Neural Networks: Neural Diffusion and Beyond](https://mlsanity.com/paper/2310.10121/) (2023)
- [ResNet-LDDMM: Advancing the LDDMM Framework using Deep Residual Networks](https://mlsanity.com/paper/2102.07951/) (2021)
- [Algorithmic Stability of Heavy-Tailed Stochastic Gradient Descent on Least Squares](https://mlsanity.com/paper/2206.01274/) (2022)
- [Second-Order Neural ODE Optimizer](https://mlsanity.com/paper/2109.14158/) (2021)
- [Lipschitz constant estimation for general neural network architectures using control tools](https://mlsanity.com/paper/2405.01125/) (2024)
- [Towards a Systems Theory of Algorithms](https://mlsanity.com/paper/2401.14029/) (2024)
- [Kullback-Leibler control for discrete-time nonlinear systems on continuous spaces](https://mlsanity.com/paper/2203.12864/) (2022)
- [Tilted Sharpness-Aware Minimization](https://mlsanity.com/paper/2410.22656/) (2024)
- [Optimal learning rate schedules in high-dimensional non-convex optimization problems](https://mlsanity.com/paper/2202.04509/) (2022)
- [PID Control-Based Self-Healing to Improve the Robustness of Large Language Models](https://mlsanity.com/paper/2404.00828/) (2024)
- [Rethinking the Relationship between Recurrent and Non-Recurrent Neural Networks: A Study in Sparsity](https://mlsanity.com/paper/2404.00880/) (2024)
- [Neural Predictive Control to Coordinate Discrete- and Continuous-Time Models for Time-Series Analysis with Control-Theoretical Improvements](https://mlsanity.com/paper/2508.01833/) (2025)
- [Generative Adversarial Network: Some Analytical Perspectives](https://mlsanity.com/paper/2104.12210/) (2021)
- [Dynamic Game Theoretic Neural Optimizer](https://mlsanity.com/paper/2105.03788/) (2021)
- [Blending Optimal Control and Biologically Plausible Learning for Noise-Robust Physical Neural Networks](https://mlsanity.com/paper/2502.19053/) (2025)
- [There is a Singularity in the Loss Landscape](https://mlsanity.com/paper/2201.06964/) (2022)
- [Asymptotically Fair Participation in Machine Learning Models: an Optimal Control Perspective](https://mlsanity.com/paper/2311.10223/) (2023)
- [Kuramoto Oscillators and Swarms on Manifolds for Geometry Informed Machine Learning](https://mlsanity.com/paper/2405.09453/) (2024)
- [Adversarial Data Augmentation for Single Domain Generalization via Lyapunov Exponent-Guided Optimization](https://mlsanity.com/paper/2507.04302/) (2025)
- [Scrap Your Schedules with PopDescent](https://mlsanity.com/paper/2310.14671/) (2023)
- [Improving Robustness via Tilted Exponential Layer: A Communication-Theoretic Perspective](https://mlsanity.com/paper/2311.01047/) (2023)
- [A Statistical Framework for Model Selection in LSTM Networks](https://mlsanity.com/paper/2506.06840/) (2025)
- [The Hamilton-Jacobi Theory of Deep Learning](https://mlsanity.com/paper/2605.28983/) (2026)
- [Generalization Bounds on Optimal Control for Transformer Training and Wasserstein Distributional Robustness](https://mlsanity.com/paper/2607.27975/) (2026)
- [A unified framework for global and local interpretability using adaptive derivative-ordered random explanation](https://mlsanity.com/paper/2609.17171/) (2026)

## Abstract (arXiv)

Attempts from different disciplines to provide a fundamental understanding of deep learning have advanced rapidly in recent years, yet a unified framework remains relatively limited. In this article, we provide one possible way to align existing branches of deep learning theory through the lens of dynamical system and optimal control. By viewing deep neural networks as discrete-time nonlinear dynamical systems, we can analyze how information propagates through layers using mean field theory. When optimization algorithms are further recast as controllers, the ultimate goal of training processes can be formulated as an optimal control problem. In addition, we can reveal convergence and generalization properties by studying the stochastic dynamics of optimization algorithms. This viewpoint features a wide range of theoretical study from information bottleneck to statistical physics. It also provides a principled way for hyper-parameter tuning when optimal control theory is introduced. Our framework fits nicely with supervised learning and can be extended to other learning problems, such as Bayesian learning, adversarial training, and specific forms of meta learning, without efforts. The review aims to shed lights on the importance of dynamics and optimal control when developing deep learning theory.