# Neural Actor-Critic Methods for Hamilton-Jacobi-Bellman PDEs: Asymptotic Analysis and Numerical Studies

arXiv: https://arxiv.org/abs/2507.06428 · HTML: https://mlsanity.com/paper/2507.06428/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 8 Jul 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 20 of 20)

- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space](https://mlsanity.com/paper/2005.05409/) (2020)
- [Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations](https://mlsanity.com/paper/1709.05963/) (2017)
- [Deep Learning Approximation for Stochastic Control Problems](https://mlsanity.com/paper/1611.07422/) (2016)
- [Actor-Critic Method for High Dimensional Static Hamilton--Jacobi--Bellman Partial Differential Equations based on Neural Networks](https://mlsanity.com/paper/2102.11379/) (2021)
- [Deep neural networks algorithms for stochastic control problems on finite horizon: convergence analysis](https://mlsanity.com/paper/1812.04300/) (2018)
- [Recent Developments in Machine Learning Methods for Stochastic Control and Games](https://mlsanity.com/paper/2303.10257/) (2023)
- [Deep Learning for Mean Field Games and Mean Field Control with Applications to Finance](https://mlsanity.com/paper/2107.04568/) (2021)
- [Regularity and stability of feedback relaxed controls](https://mlsanity.com/paper/2001.03148/) (2020)
- [Extensions of the Deep Galerkin Method](https://mlsanity.com/paper/1912.01455/) (2019)
- [Mean-field neural networks-based algorithms for McKean-Vlasov control problems *](https://mlsanity.com/paper/2212.11518/) (2022)
- [A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guarantee](https://mlsanity.com/paper/2302.05816/) (2023)
- [Global Convergence of Deep Galerkin and PINNs Methods for Solving Partial Differential Equations](https://mlsanity.com/paper/2305.06000/) (2023)
- [Neural Q-learning for solving PDEs](https://mlsanity.com/paper/2203.17128/) (2022)
- [Deep Learning for Continuous-Time Stochastic Control with Jumps](https://mlsanity.com/paper/2505.15602/) (2025)
- [Convergence of the Deep Galerkin Method for Finite State Mean Field Control Problems](https://mlsanity.com/paper/2405.13346/) (2024)
- [Continuous Policy and Value Iteration for Relaxed Stochastic Control Problems and Its Convergence](https://mlsanity.com/paper/2506.08121/) (2025)

## Built on by (most cited first, 3 of 3)

- [Random Gradient-Free Optimization in Infinite Dimensional Spaces](https://mlsanity.com/paper/2512.20566/) (2025)
- [DeepPAAC: A New Deep Galerkin Method for Principal-Agent Problems](https://mlsanity.com/paper/2511.04309/) (2025)
- [Policy Gradient for Continuous-Time Robust Markov Decision Processes](https://mlsanity.com/paper/2606.04335/) (2026)

## Abstract (arXiv)

We mathematically analyze and numerically study an actor-critic machine learning algorithm for solving high-dimensional Hamilton-Jacobi-Bellman (HJB) partial differential equations from stochastic control theory. The architecture of the critic (the estimator for the value function) is structured so that the boundary condition is always perfectly satisfied (rather than being included in the training loss) and utilizes a biased gradient which reduces computational cost. The actor (the estimator for the optimal control) is trained by minimizing the integral of the Hamiltonian over the domain, where the Hamiltonian is estimated using the critic. We show that the training dynamics of the actor and critic neural networks converge in a Sobolev-type space to a certain infinite-dimensional ordinary differential equation (ODE) as the number of hidden units in the actor and critic rightarrow ∞. Further, under a convexity-like assumption on the Hamiltonian, we prove that any fixed point of this limit ODE is a solution of the original stochastic control problem. This provides an important guarantee for the algorithm's performance in light of the fact that finite-width neural networks may only converge to a local minimizers (and not optimal solutions) due to the non-convexity of their loss functions. In our numerical studies, we demonstrate that the algorithm can solve stochastic control problems accurately in up to 200 dimensions. In particular, we construct a series of increasingly complex stochastic control problems with known analytic solutions and study the algorithm's numerical performance on them. These problems range from a linear-quadratic regulator equation to highly challenging equations with non-convex Hamiltonians, allowing us to identify and analyze the strengths and limitations of this neural actor-critic method for solving HJB equations.