mlsanityData to 14 Oct 2026

Neural Actor-Critic Methods for Hamilton-Jacobi-Bellman PDEs: Asymptotic Analysis and Numerical Studies

arXiv 2507.06428 ↗First posted 8 Jul 2025math.OC cs.LG cs.NAcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We mathematically analyze and numerically study an actor-critic machine learning algorithm for solving high-dimensional Hamilton-Jacobi-Bellman (HJB) partial differential equations from stochastic control theory. The architecture of the critic (the estimator for the value function) is structured so that the boundary condition is always perfectly satisfied (rather than being included in the training loss) and utilizes a biased gradient which reduces computational cost. The actor (the estimator for the optimal control) is trained by minimizing the integral of the Hamiltonian over the domain, where the Hamiltonian is estimated using the critic. We show that the training dynamics of the actor and critic neural networks converge in a Sobolev-type space to a certain infinite-dimensional ordinary differential equation (ODE) as the number of hidden units in the actor and critic rightarrow ∞. Further, under a convexity-like assumption on the Hamiltonian, we prove that any fixed point of this limit ODE is a solution of the original stochastic control problem. This provides an important guarantee for the algorithm's performance in light of the fact that finite-width neural networks may only converge to a local minimizers (and not optimal solutions) due to the non-convexity of their loss functions. In our numerical studies, we demonstrate that the algorithm can solve stochastic control problems accurately in up to 200 dimensions. In particular, we construct a series of increasingly complex stochastic control problems with known analytic solutions and study the algorithm's numerical performance on them. These problems range from a linear-quadratic regulator equation to highly challenging equations with non-convex Hamiltonians, allowing us to identify and analyze the strengths and limitations of this neural actor-critic method for solving HJB equations.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 37 references that are in the map. First posted on arXiv 8 Jul 2025.

Builds on

20 of 37 references.

References · 20

  1. 2017DGM: A deep learning algorithm for solving partial differential equationscited by 678
  2. 2017Solving high-dimensional partial differential equations using deep learningcited by 556
  3. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  4. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  5. 2020Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path spacecited by 82
  6. 2017Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equationscited by 65
  7. 2016Deep Learning Approximation for Stochastic Control Problemscited by 55
  8. 2021Actor-Critic Method for High Dimensional Static Hamilton--Jacobi--Bellman Partial Differential Equations based on Neural Networkscited by 29
  9. 2018Deep neural networks algorithms for stochastic control problems on finite horizon: convergence analysiscited by 24
  10. 2023Recent Developments in Machine Learning Methods for Stochastic Control and Gamescited by 24
  11. 2021Deep Learning for Mean Field Games and Mean Field Control with Applications to Financecited by 20
  12. 2020Regularity and stability of feedback relaxed controlscited by 13
8 more
  1. 2019Extensions of the Deep Galerkin Methodcited by 10
  2. 2022Mean-field neural networks-based algorithms for McKean-Vlasov control problems *cited by 10
  3. 2023A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guaranteecited by 10
  4. 2023Global Convergence of Deep Galerkin and PINNs Methods for Solving Partial Differential Equationscited by 10
  5. 2022Neural Q-learning for solving PDEscited by 8
  6. 2025Deep Learning for Continuous-Time Stochastic Control with Jumpscited by 6
  7. 2024Convergence of the Deep Galerkin Method for Finite State Mean Field Control Problemscited by 2
  8. 2025Continuous Policy and Value Iteration for Relaxed Stochastic Control Problems and Its Convergencecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Random Gradient-Free Optimization in Infinite Dimensional Spacescited by 1
  2. 2025DeepPAAC: A New Deep Galerkin Method for Principal-Agent Problemscited by 0
  3. 2026Policy Gradient for Continuous-Time Robust Markov Decision Processescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Viscosity-Informed Generative Actor-Critic for High-Dimensional Stochastic Optimal Control
  2. 2024Weak Convergence Analysis of Online Neural Actor-Critic Algorithms
  3. 2024Actor-Critic Physics-informed Neural Lyapunov Control
  4. 2026An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flows
  5. 2025Solving nonconvex Hamilton--Jacobi--Isaacs equations with PINN-based policy iteration
  6. 2026From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments
  7. 2026Refined Analysis of Entropy-Regularized Actor-Critic
  8. 2024Neural Network Approaches for Parameterized Optimal Control