mlsanityData to 14 Oct 2026

Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning

arXiv 2503.09981 ↗First posted 13 Mar 2025cs.LG cs.NA math.NAcited by 13

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Stochastic policies (also known as relaxed controls) are widely used in continuous-time reinforcement learning algorithms. However, executing a stochastic policy and evaluating its performance in a continuous-time environment remain open challenges. This work introduces and rigorously analyzes a policy execution framework that samples actions from a stochastic policy at discrete time points and implements them as piecewise constant controls. We prove that as the sampling mesh size tends to zero, the controlled state process converges weakly to the dynamics with coefficients aggregated according to the stochastic policy. We explicitly quantify the convergence rate based on the regularity of the coefficients and establish an optimal first-order convergence rate for sufficiently regular coefficients. Additionally, we prove a 1/2-order weak convergence rate that holds uniformly over the sampling noise with high probability, and establish a 1/2-order pathwise convergence for each realization of the system noise in the absence of volatility control. Building on these results, we analyze the bias and variance of various policy evaluation and policy gradient estimators based on discrete-time observations. Our results provide theoretical justification for the exploratory stochastic control framework in [H. Wang, T. Zariphopoulou, and X.Y. Zhou, J. Mach. Learn. Res., 21 (2020), pp. 1-34].

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 18 of its 50 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

18 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024On the grid-sampling limit SDE10 Oct
  2. 2024PhiBE: A PDE-based Bellman Equation for Continuous Time Policy Evaluation21 May
  3. 2024Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty19 Apr

Other references · 15

  1. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  2. 2022q-Learning in Continuous Timecited by 91
  3. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 84
  4. 2019Making Deep Q-learning methods robust to time discretizationcited by 58
  5. 2020Entropy Regularization for Mean Field Games with Learningcited by 58
  6. 2021Continuous-Time Model-Based Reinforcement Learningcited by 41
  7. 2020Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizoncited by 35
  8. 2020Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controlscited by 31
  9. 2023Policy Optimization for Continuous Reinforcement Learningcited by 31
  10. 2022Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problemscited by 25
  11. 2021Reinforcement learning for linear-convex models with jumps via stability analysis of feedback controlscited by 24
  12. 2023Actor-Critic learning for mean-field control in continuous timecited by 22
3 more
  1. 2023Continuous-time q-learning for mean-field control problemscited by 22
  2. 2024Entropy annealing for policy mirror descent in continuous time and spacecited by 12
  3. 2024Convergence Analysis for Entropy-Regularized Control Problems: A Probabilistic Approachcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 11.

13 papers.

Sort:

By area:

  1. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and Statecited by 8
  2. 2026Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundationscited by 6
  3. 2026Policy Gradient for Continuous-Time Mean-Field Controlcited by 4
  4. 2025Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Controlcited by 3
  5. 2026Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problemscited by 3
  6. 2026Discretization error from regularized Reinforcement Learning to continuous-time stochastic controlcited by 2
  7. 2026Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Datacited by 1
  8. 2026Actor-Critic Learning for Extended Mean Field Control with Deterministic Policiescited by 1
  9. 2026Continuous-time reinforcement learning: ellipticity enables model-free value function approximationcited by 0
  10. 2026An Actor-Critic Framework for Continuous-Time Jump-Diffusion Controls with Normalizing Flowscited by 0
  11. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learningcited by 0
  12. 2026Convergence and Regret of the Policy Gradient for Multi-Armed Bandits in Diffusion Environmentcited by 0
1 more
  1. 2026Amortized Score-Hamiltonian Policy Iteration: A Grid-Free Scheme for Relaxed Stochastic Control Problemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Equivalence of stochastic and deterministic policy gradients
  2. 2026Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence
  3. 2024A random measure approach to reinforcement learning in continuous time
  4. 2025Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation
  5. 2024Learning Optimal Deterministic Policies with Stochastic Policy Gradients
  6. 2026Policy Gradient for Continuous-Time Robust Markov Decision Processes
  7. 2024Reusing Historical Trajectories in Natural Policy Gradient via Importance Sampling: Convergence and Convergence Rate
  8. 2026Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients