mlsanityData to 14 Oct 2026

A hierarchical resource-efficient deep policy gradient method for continuous-time optimal control problems

arXiv 2502.14141 ↗First posted 19 Feb 2025math.OC q-fin.CPcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In this paper, we propose an efficient implementation of a deep policy gradient method (PGM) for optimal control problems in continuous time. For continuous-time problems that require a fine time discretization to achieve a desired accuracy, the proposed method improves time efficiency and performance by strategically allocating computational resources across scales, i.e., the number of trajectories, the granularity of time discretization, and the complexity of the neural network architecture. The main idea of the paper is to avoid committing to a fine time discretization. At first, we train a policy, modeled by a neural network, for a discretized optimal control problem in a coarse time scale. Then, we only discretize more in the time intervals where there are indications that the coarse scheme is not accurate enough and train a new policy in the finer scale. We then proceed to refine the time grid further to achieve better accuracy in the new smaller time intervals. Our theoretical result indicates how the new schedule for allocation of resources in different time scales can lead to efficiency. We conclude the paper by numerical experiments on a linear-quadratic stochastic optimal control problem and an optimal execution problem from quantitative finance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 21 of its 46 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

21 of 46 references.

References · 21

  1. 2017DGM: A deep learning algorithm for solving partial differential equationscited by 678
  2. 2017Solving high-dimensional partial differential equations using deep learningcited by 556
  3. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 243
  4. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  5. 2022q-Learning in Continuous Timecited by 91
  6. 2017A Hitting Time Analysis of Stochastic Gradient Langevin Dynamicscited by 71
  7. 2017Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equationscited by 65
  8. 2016Deep Learning Approximation for Stochastic Control Problemscited by 55
  9. 2020Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizoncited by 50
  10. 2020Non-convex Learning via Replica Exchange Stochastic Gradient MCMCcited by 34
  11. 2022Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problemscited by 25
  12. 2016Algorithms for Overcoming the Curse of Dimensionality for Certain Hamilton-Jacobi Equations Arising in Control Theory and Elsewherecited by 24
9 more
  1. 2019Neural networks-based backward scheme for fully nonlinear PDEscited by 23
  2. 2021Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methodscited by 23
  3. 2020Approximation error analysis of some deep backward schemes for nonlinear PDEscited by 21
  4. 2020Accelerating Nonconvex Learning via Replica Exchange Langevin Diffusioncited by 16
  5. 2023A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guaranteecited by 10
  6. 2017Perspectives on characteristics based curse-of-dimensionality-free numerical approaches for solving Hamilton-Jacobi equationscited by 9
  7. 2022Linear convergence of a policy gradient method for some finite horizon continuous time control problemscited by 7
  8. 2020Deep Empirical Risk Minimization in finance: looking into the futurecited by 6
  9. 2022On Parametric Optimal Execution and Machine Learning Surrogatescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024An Efficient On-Policy Deep Learning Framework for Stochastic Optimal Control
  2. 2024The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations
  3. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  4. 2024Identifying Policy Gradient Subspaces
  5. 2024PWM: Policy Learning with Multi-Task World Models
  6. 2024Neural Network Approaches for Parameterized Optimal Control
  7. 2026Continuous-time Optimal Stopping through Deep Reinforcement Learning
  8. 2025Convergence of Proximal Policy Gradient Method for Problems with Control Dependent Diffusion Coefficients