mlsanityData to 14 Oct 2026

Optimal Control Theoretic Neural Optimizer: From Backpropagation to Dynamic Programming

arXiv 2510.14168 ↗First posted 15 Oct 2025cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Optimization of deep neural networks (DNNs) has been a driving force in the advancement of modern machine learning and artificial intelligence. With DNNs characterized by a prolonged sequence of nonlinear propagation, determining their optimal parameters given an objective naturally fits within the framework of Optimal Control Programming. Such an interpretation of DNNs as dynamical systems has proven crucial in offering a theoretical foundation for principled analysis from numerical equations to physics. In parallel to these theoretical pursuits, this paper focuses on an algorithmic perspective. Our motivated observation is the striking algorithmic resemblance between the Backpropagation algorithm for computing gradients in DNNs and the optimality conditions for dynamical systems, expressed through another backward process known as dynamic programming. Consolidating this connection, where Backpropagation admits a variational structure, solving an approximate dynamic programming up to the first-order expansion leads to a new class of optimization methods exploring higher-order expansions of the Bellman equation. The resulting optimizer, termed Optimal Control Theoretic Neural Optimizer (OCNOpt), enables rich algorithmic opportunities, including layer-wise feedback policies, game-theoretic applications, and higher-order training of continuous-time models such as Neural ODEs. Extensive experiments demonstrate that OCNOpt improves upon existing methods in robustness and efficiency while maintaining manageable computational complexity, paving new avenues for principled algorithmic design grounded in dynamical systems and optimal control theory.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 61 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

32 of 61 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  4. 2017Mask R-CNNcited by 7,277
  5. 2018Neural Ordinary Differential Equationscited by 3,779
  6. 2016Geometric deep learning: going beyond Euclidean datacited by 1,048
  7. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  8. 2019Hamiltonian Neural Networkscited by 712
  9. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 594
  10. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 506
  11. 2014New insights and perspectives on the natural gradient methodcited by 443
  12. 2018The UEA multivariate time series classification archive, 2018cited by 338
20 more
  1. 2020Scalable Gradients for Stochastic Differential Equationscited by 271
  2. 2021Likelihood Training of Schr"odinger Bridge using Forward-Backward SDEs Theorycited by 219
  3. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  4. 2019You Only Propagate Once: Accelerating Adversarial Training via Maximal Principlecited by 183
  5. 2019Latent ODEs for Irregularly-Sampled Time Seriescited by 183
  6. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 146
  7. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  8. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 142
  9. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
  10. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
  11. 2017Maximum Principle Based Algorithms for Deep Learningcited by 78
  12. 2017Reversible Architectures for Arbitrarily Deep Residual Neural Networkscited by 70
  13. 2018A Mean-Field Optimal Control Formulation of Deep Learningcited by 67
  14. 2019Pontryagin Differentiable Programming: An End-to-End Learning and Control Frameworkcited by 52
  15. 2022Deep Generalized Schr"odinger Bridgecited by 47
  16. 2017OL'E: Orthogonal Low-rank Embedding, A Plug and Play Geometric Loss for Deep Learningcited by 37
  17. 2018Layer-Parallel Training of Deep Residual Neural Networkscited by 35
  18. 2018An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networkscited by 26
  19. 2019Cross-Entropy Loss and Low-Rank Features Have Responsibility for Adversarial Examplescited by 17
  20. 2020Robust Deep Learning as Optimal Control: Insights and Convergence Guaranteescited by 7

Built on by

Not cited yet.