# A hierarchical resource-efficient deep policy gradient method for continuous-time optimal control problems

arXiv: https://arxiv.org/abs/2502.14141 · HTML: https://mlsanity.com/paper/2502.14141/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 46 references that are in the map.
- First posted on arXiv 19 Feb 2025.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 21 of 21)

- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis](https://mlsanity.com/paper/1702.03849/) (2017)
- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [A Hitting Time Analysis of Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/1702.05575/) (2017)
- [Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations](https://mlsanity.com/paper/1709.05963/) (2017)
- [Deep Learning Approximation for Stochastic Control Problems](https://mlsanity.com/paper/1611.07422/) (2016)
- [Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon](https://mlsanity.com/paper/2011.10300/) (2020)
- [Non-convex Learning via Replica Exchange Stochastic Gradient MCMC](https://mlsanity.com/paper/2008.05367/) (2020)
- [Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems](https://mlsanity.com/paper/2211.00617/) (2022)
- [Algorithms for Overcoming the Curse of Dimensionality for Certain Hamilton-Jacobi Equations Arising in Control Theory and Elsewhere](https://mlsanity.com/paper/1605.01799/) (2016)
- [Neural networks-based backward scheme for fully nonlinear PDEs](https://mlsanity.com/paper/1908.00412/) (2019)
- [Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methods](https://mlsanity.com/paper/2111.03941/) (2021)
- [Approximation error analysis of some deep backward schemes for nonlinear PDEs](https://mlsanity.com/paper/2006.01496/) (2020)
- [Accelerating Nonconvex Learning via Replica Exchange Langevin Diffusion](https://mlsanity.com/paper/2007.01990/) (2020)
- [A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guarantee](https://mlsanity.com/paper/2302.05816/) (2023)
- [Perspectives on characteristics based curse-of-dimensionality-free numerical approaches for solving Hamilton-Jacobi equations](https://mlsanity.com/paper/1711.03314/) (2017)
- [Linear convergence of a policy gradient method for some finite horizon continuous time control problems](https://mlsanity.com/paper/2203.11758/) (2022)
- [Deep Empirical Risk Minimization in finance: looking into the future](https://mlsanity.com/paper/2011.09349/) (2020)
- [On Parametric Optimal Execution and Machine Learning Surrogates](https://mlsanity.com/paper/2204.08581/) (2022)

## Abstract (arXiv)

In this paper, we propose an efficient implementation of a deep policy gradient method (PGM) for optimal control problems in continuous time. For continuous-time problems that require a fine time discretization to achieve a desired accuracy, the proposed method improves time efficiency and performance by strategically allocating computational resources across scales, i.e., the number of trajectories, the granularity of time discretization, and the complexity of the neural network architecture. The main idea of the paper is to avoid committing to a fine time discretization. At first, we train a policy, modeled by a neural network, for a discretized optimal control problem in a coarse time scale. Then, we only discretize more in the time intervals where there are indications that the coarse scheme is not accurate enough and train a new policy in the finer scale. We then proceed to refine the time grid further to achieve better accuracy in the new smaller time intervals. Our theoretical result indicates how the new schedule for allocation of resources in different time scales can lead to efficiency. We conclude the paper by numerical experiments on a linear-quadratic stochastic optimal control problem and an optimal execution problem from quantitative finance.