mlsanityData to 14 Oct 2026

A random measure approach to reinforcement learning in continuous time

arXiv 2409.17200 ↗First posted 25 Sep 2024cs.LG math.PR stat.MLcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We present a random measure approach for modeling exploration, i.e., the execution of measure-valued controls, in continuous-time reinforcement learning (RL) with controlled diffusion and jumps. First, we consider the case when sampling the randomized control in continuous time takes place on a discrete-time grid and reformulate the resulting stochastic differential equation (SDE) as an equation driven by suitable random measures. The construction of these random measures makes use of the Brownian motion and the Poisson random measure (which are the sources of noise in the original model dynamics) as well as the additional random variables, which are sampled on the grid for the control execution. Then, we prove a limit theorem for these random measures as the mesh-size of the sampling grid goes to zero, which leads to the grid-sampling limit SDE that is jointly driven by white noise random measures and a Poisson random measure. We also argue that the grid-sampling limit SDE can substitute the exploratory SDE and the sample SDE of the recent continuous-time RL literature, i.e., it can be applied for the theoretical analysis of exploratory control problems and for the derivation of learning algorithms.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 9 of its 34 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

9 of 34 references.

References · 9

  1. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  2. 2022q-Learning in Continuous Timecited by 91
  3. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 84
  4. 2020Entropy Regularization for Mean Field Games with Learningcited by 58
  5. 2021Exploratory HJB equations and their convergencecited by 32
  6. 2022Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problemscited by 25
  7. 2020Exploratory LQG Mean Field Games with Entropy Regularizationcited by 15
  8. 2020Regularity and stability of feedback relaxed controlscited by 13
  9. 2023Entropy-Regularized Mean-Variance Portfolio Optimization with Jumpscited by 5

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024On the grid-sampling limit SDEcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning
  2. 2025Sample-efficient and Scalable Exploration in Continuous-Time RL
  3. 2026From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments
  4. 2025Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control
  5. 2026Q-Measure-Learning for Continuous State RL: Efficient Implementation and Convergence
  6. 2024Control randomisation approach for policy gradient and application to reinforcement learning in optimal switching
  7. 2026Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence
  8. 2025Reinforcement Learning with Random Time Horizons