mlsanityData to 7 Oct 2026

A Mean-Field Optimal Control Formulation of Deep Learning

arXiv 1807.01083 ↗First posted 3 Jul 2018math.OC cs.LGcited by 67

Not on the map. Why?

Abstract (arXiv)

Recent work linking deep neural networks and dynamical systems opened up new avenues to analyze deep learning. In particular, it is observed that new insights can be obtained by recasting deep learning as an optimal control problem on difference or differential equations. However, the mathematical aspects of such a formulation have not been systematically explored. This paper introduces the mathematical formulation of the population risk minimization problem in deep learning as a mean-field optimal control problem. Mirroring the development of classical optimal control, we state and prove optimality conditions of both the Hamilton-Jacobi-Bellman type and the Pontryagin type. These mean-field results reflect the probabilistic nature of the learning problem. In addition, by appealing to the mean-field Pontryagin's maximum principle, we establish some quantitative relationships between population and empirical learning problems. This serves to establish a mathematical foundation for investigating the algorithmic and theoretical connections between optimal control and deep learning.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 12; 2023: 11; 2024: 11; 2025: 8; 2026: 12.

67 papers.

Sort:

  1. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  2. 2023The emergence of clusters in self-attention dynamicscited by 133
  3. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 54
  4. 2022Deep Generalized Schr"odinger Bridgecited by 47
  5. 2024Data Selection via Optimal Control for Language Modelscited by 38
  6. 2021Momentum Residual Neural Networkscited by 36
  7. 2021Bayesian Uncertainty Estimation of Learned Variational MRI Reconstructioncited by 30
  8. 2022Do Residual Neural Networks discretize Neural Ordinary Differential Equations?cited by 27
  9. 2021Neural Delay Differential Equationscited by 26
  10. 2025The Mean-Field Dynamics of Transformerscited by 24
  11. 2021Towards Robust Neural Networks via Close-loop Controlcited by 23
  12. 2022On Numerical Integration in Neural Ordinary Differential Equationscited by 21
38 more
  1. 2024Operator Learning Using Random Features: A Tool for Scientific Computingcited by 21
  2. 2023Implicit regularization of deep residual networks towards neural ODEscited by 20
  3. 2022Scaling ResNets in the Large-depth Regimecited by 18
  4. 2021Scaling Properties of Deep Residual Networkscited by 16
  5. 2023Interpolation, Approximation and Controllability of Deep Neural Networkscited by 15
  6. 2024Optimal Protocols for Continual Learning via Statistical Physics and Control Theorycited by 14
  7. 2021Second-Order Neural ODE Optimizercited by 11
  8. 2021On the regularized risk of distributionally robust learning over deep neural networkscited by 8
  9. 2024Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transportcited by 7
  10. 2025Universal Approximation Theorem of Deep Q-Networkscited by 6
  11. 2021Neural network architectures using min-plus algebra for solving certain high dimensional optimal control problems and Hamilton-Jacobi PDEscited by 5
  12. 2022Why Quantization Improves Generalization: NTK of Binary Weight Neural Networkscited by 5
  13. 2024Physics-informed Neural Networks for Functional Differential Equations: Cylindrical Approximation and Its Convergence Guaranteescited by 5
  14. 2021Approximation capabilities of measure-preserving neural networkscited by 4
  15. 2022Neural Piecewise-Constant Delay Differential Equationscited by 4
  16. 2023Cluster-based classification with neural ODEs via controlcited by 4
  17. 2024WGFormer: An SE(3)-Transformer Driven by Wasserstein Gradient Flows for Molecular Ground-State Conformation Predictioncited by 4
  18. 2022Asymptotic Analysis of Deep Residual Networkscited by 3
  19. 2023Variational formulations of ODE-Net as a mean-field optimal control problem and existence resultscited by 3
  20. 2023From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networkscited by 3
  21. 2024PID Control-Based Self-Healing to Improve the Robustness of Large Language Modelscited by 3
  22. 2024Rethinking the Relationship between Recurrent and Non-Recurrent Neural Networks: A Study in Sparsitycited by 3
  23. 2024Online Control-Informed Learningcited by 3
  24. 2025Neural Predictive Control to Coordinate Discrete- and Continuous-Time Models for Time-Series Analysis with Control-Theoretical Improvementscited by 3
  25. 2026An Optimal Control Approach To Transformer Trainingcited by 3
  26. 2021Dynamic Game Theoretic Neural Optimizercited by 2
  27. 2021Neural Network Layer Algebra: A Framework to Measure Capacity and Compression in Deep Learningcited by 2
  28. 2022The Mori-Zwanzig formulation of deep learningcited by 2
  29. 2022Continuous-in-time Limit for Bayesian Banditscited by 2
  30. 2025Universal Approximation Theorem for Deep Q-Learning via FBSDE Systemcited by 2
  31. 2025Geometric Foundations of Tuning without Forgetting in Neural ODEscited by 2
  32. 2026GenCtrl -- A Formal Controllability Toolkit for Generative Modelscited by 2
  33. 2023Deep Learning for Mean Field Games with non-separable Hamiltonianscited by 1
  34. 2023Efficient Quantum Algorithms for Quantum Optimal Controlcited by 1
  35. 2023Asymptotically Fair Participation in Machine Learning Models: an Optimal Control Perspectivecited by 1
  36. 2026Understanding the Theoretical Foundations of Deep Neural Networks through Differential Equationscited by 1
  37. 2026Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformerscited by 1
  38. 2022Physical Derivatives: Computing policy gradients by physical forward-propagationcited by 0

50 most cited of 67 All 67 (JSON)