mlsanityData to 14 Oct 2026

Policy Gradient for Continuous-Time Robust Markov Decision Processes

arXiv 2606.04335 ↗First posted 3 Jun 2026cs.LG cs.SY eess.SYcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

The framework of robust Markov decision processes (RMDPs) allows the design of reinforcement learning agents that satisfy performance guarantees under worst-case transition dynamics. Traditional RMDPs consider discrete-time dynamics and recently, sample-efficient policy gradient algorithms have been considered in this context. This paper investigates policy gradient algorithms within a continuous-time RMDP framework. Policy gradients and adversarial gradients are derived using pathwise and adjoint-based formulas for stochastic and ordinary differential equations. We propose double-loop optimisers to obtain linear convergence in the oracle-based setting and an tildeO(frac1ε2) sample complexity in the sample-based setting in an analysis which also derives novel tools for the framework of undiscounted total cost MDPs. Additionally, we propose mean-field optimisers as distributional optimisers with an tildeO(frac1K) oracle-based convergence rate and an tildeO(fracN2ε) sample complexity under N-particle approximation. The effectiveness of continuous-time policy gradient algorithms is confirmed for both optimisers on continuous-time RMDPs with neural ordinary differential equation dynamics.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 36 of its 56 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

36 of 56 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Why Policy Gradient Algorithms Work for Undiscounted Total-Reward MDPs21 Oct
  2. 2025Neural Actor-Critic Methods for Hamilton-Jacobi-Bellman PDEs: Asymptotic Analysis and Numerical Studies8 Jul
  3. 2025Mirror Descent Policy Optimisation for Robust Constrained Markov Decision Processes29 Jun
  4. 2025Wasserstein Policy Optimization1 May
  5. 2025Optimal Control of Probabilistic Dynamics Models via Mean Hamiltonian Minimization3 Apr
  6. 2024Policy Gradient for Robust Markov Decision Processes29 Oct
  7. 2024Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients28 Mar
  8. 2024Stabilizing Policy Gradients for Stochastic Differential Equations via Consistency with Perturbation Process7 Mar

Other references · 28

  1. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  2. 2018Neural Ordinary Differential Equationscited by 3,779
  3. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  4. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  5. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  6. 2019Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limitcited by 176
  7. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  8. 2017Maximum Principle Based Algorithms for Deep Learningcited by 78
  9. 2019Pontryagin Differentiable Programming: An End-to-End Learning and Control Frameworkcited by 52
  10. 2018Finding Mixed Nash Equilibria of Generative Adversarial Networkscited by 49
  11. 2020Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controlscited by 31
  12. 2023Policy Optimization for Continuous Reinforcement Learningcited by 31
16 more
  1. 2022Uniform-in-time propagation of chaos for mean field Langevin dynamicscited by 23
  2. 2023A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spacescited by 23
  3. 2023Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reductioncited by 20
  4. 2023Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Setscited by 16
  5. 2016Stochastic Gradient Descent in Continuous Timecited by 14
  6. 2022Two-Scale Gradient Descent Ascent Dynamics Finds Mixed Nash Equilibria of Continuous Games: A Mean-Field Perspectivecited by 13
  7. 2023A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guaranteecited by 10
  8. 2023How to Learn and Generalize From Three Minutes of Data: Physics-Constrained and Uncertainty-Aware Neural Stochastic Differential Equationscited by 10
  9. 2020Faster Policy Learning with Continuous-Time Gradientscited by 8
  10. 2023Distributionally Robust Linear Quadratic Controlcited by 8
  11. 2019Gradient projection and conditional gradient methods for constrained nonconvex minimizationcited by 4
  12. 2024Risk-averse Total-reward MDPs with ERM and EVaRcited by 4
  13. 2022First-Order Pontryagin Maximum Principle for Risk-Averse Stochastic Optimal Control Problemscited by 3
  14. 2025Mirror Mean-Field Langevin Dynamicscited by 3
  15. 2026Revisiting Subgradient Dominance in Robust MDPs: Counterexamples, Hardness, and Sufficient Conditionscited by 1
  16. 2026Robust Markov Decision Processes on Continuous State Spacescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Single-Loop Robust Policy Gradient Method for Robust Markov Decision Processes
  2. 2025Policy Regularized Distributionally Robust Markov Decision Processes with Linear Function Approximation
  3. 2024Strongly-polynomial time and validation analysis of policy gradient methods
  4. 2024Robust Markov Decision Processes: A Place Where AI and Formal Methods Meet
  5. 2025Provably Efficient Sample Complexity for Robust CMDP
  6. 2024Sequential Decision-Making under Uncertainty: A Robust MDPs review
  7. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  8. 2024Landscape of Policy Optimization for Finite Horizon MDPs with General State and Action