# A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guarantee

arXiv: https://arxiv.org/abs/2302.05816 · HTML: https://mlsanity.com/paper/2302.05816/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 55 references that are in the map.
- First posted on arXiv 11 Feb 2023.

## Builds on (most cited first, 22 of 22)

- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Condition](https://mlsanity.com/paper/1608.04636/) (2016)
- [A Finite Time Analysis of Two Time-Scale Actor Critic Methods](https://mlsanity.com/paper/2005.01350/) (2020)
- [A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problems](https://mlsanity.com/paper/1912.01825/) (2019)
- [Entropy Regularization for Mean Field Games with Learning](https://mlsanity.com/paper/2010.00145/) (2020)
- [Solving high-dimensional eigenvalue problems using deep neural networks: A diffusion Monte Carlo like approach](https://mlsanity.com/paper/2002.02600/) (2020)
- [Convergence Analysis of Machine Learning Algorithms for the Numerical Solution of Mean Field Control and Games: II -- The Finite Horizon Case](https://mlsanity.com/paper/1908.01613/) (2019)
- [Exploratory HJB equations and their convergence](https://mlsanity.com/paper/2109.10269/) (2021)
- [Actor-Critic Method for High Dimensional Static Hamilton--Jacobi--Bellman Partial Differential Equations based on Neural Networks](https://mlsanity.com/paper/2102.11379/) (2021)
- [A mean-field analysis of two-player zero-sum games](https://mlsanity.com/paper/2002.06277/) (2020)
- [Deep Learning Methods for Mean Field Control Problems with Delay](https://mlsanity.com/paper/1905.00358/) (2019)
- [Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time](https://mlsanity.com/paper/2008.06845/) (2020)
- [A Neural Network Approach for High-Dimensional Optimal Control Applied to Multi-Agent Path Finding](https://mlsanity.com/paper/2104.03270/) (2021)
- [Exploratory LQG Mean Field Games with Entropy Regularization](https://mlsanity.com/paper/2011.12946/) (2020)
- [Signatured Deep Fictitious Play for Mean Field Games with Common Noise](https://mlsanity.com/paper/2106.03272/) (2021)
- [A modified MSA for stochastic control problems](https://mlsanity.com/paper/2007.05209/) (2020)
- [Gradient Flows for Regularized Stochastic Control Problems](https://mlsanity.com/paper/2006.05956/) (2020)
- [A neural network based policy iteration algorithm with global H2-superlinear convergence for stochastic games on domains](https://mlsanity.com/paper/1906.02304/) (2019)
- [Solving stochastic optimal control problem via stochastic maximum principle with deep learning method](https://mlsanity.com/paper/2007.02227/) (2020)
- [Mean field control problems for vaccine distribution](https://mlsanity.com/paper/2104.11887/) (2021)
- [Recurrent Neural Networks for Stochastic Control Problems with Delay](https://mlsanity.com/paper/2101.01385/) (2021)
- [The Modified MSA, a Gradient Flow and Convergence](https://mlsanity.com/paper/2212.05784/) (2022)

## Built on by (most cited first, 10 of 10)

- [Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence](https://mlsanity.com/paper/2412.18164/) (2024)
- [Entropy annealing for policy mirror descent in continuous time and space](https://mlsanity.com/paper/2405.20250/) (2024)
- [Neural Network Approaches for Parameterized Optimal Control](https://mlsanity.com/paper/2402.10033/) (2024)
- [Score-based Neural Ordinary Differential Equations for Computing Mean Field Control Problems](https://mlsanity.com/paper/2409.16471/) (2024)
- [Neural Actor-Critic Methods for Hamilton-Jacobi-Bellman PDEs: Asymptotic Analysis and Numerical Studies](https://mlsanity.com/paper/2507.06428/) (2025)
- [Convergence of Proximal Policy Gradient Method for Problems with Control Dependent Diffusion Coefficients](https://mlsanity.com/paper/2505.18379/) (2025)
- [Continuous Policy and Value Iteration for Relaxed Stochastic Control Problems and Its Convergence](https://mlsanity.com/paper/2506.08121/) (2025)
- [A hierarchical resource-efficient deep policy gradient method for continuous-time optimal control problems](https://mlsanity.com/paper/2502.14141/) (2025)
- [Policy Gradient for Continuous-Time Robust Markov Decision Processes](https://mlsanity.com/paper/2606.04335/) (2026)
- [Amortized Score-Hamiltonian Policy Iteration: A Grid-Free Scheme for Relaxed Stochastic Control Problems](https://mlsanity.com/paper/2610.04285/) (2026)

## Abstract (arXiv)

We consider policy gradient methods for stochastic optimal control problem in continuous time. In particular, we analyze the gradient flow for the control, viewed as a continuous time limit of the policy gradient method. We prove the global convergence of the gradient flow and establish a convergence rate under some regularity assumptions. The main novelty in the analysis is the notion of local optimal control function, which is introduced to characterize the local optimality of the iterate.