mlsanityData to 14 Oct 2026

Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control

arXiv 2504.21793 ↗First posted 30 Apr 2025math.OCcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Reinforcement learning (RL) is currently one of the most prominent methods for optimizing dynamical systems, with breakthrough results across various fields. The framework is based on the concept of a Markov decision process (MDP), leading to a discrete-time optimal control problem. In the RL literature, such problems are typically formulated and solved using mixed policies, from which random actions are sampled at each time step. Recently, part of the optimal control community has begun investigating continuous-time versions of RL algorithms, replacing MDPs with continuous-time stochastic processes governed by relaxed controls, and asserting a full analogy between the two formulations. In this work, we examine the limitations of this analogy and rigorously establish a connection between the two problems in the case where only the drift term of the continuous-time model is controlled. We prove strong convergence of the RL implementation of mixed strategies as the time discretization mesh tends to zero. We also discuss the technical challenges posed by the possible presence of control in the diffusion component of the state.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 46 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

19 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning13 Mar

Other references · 18

  1. 2021Recent Advances in Reinforcement Learning in Financecited by 110
  2. 2022q-Learning in Continuous Timecited by 91
  3. 2019Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learningcited by 59
  4. 2019Making Deep Q-learning methods robust to time discretizationcited by 58
  5. 2020Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizoncited by 35
  6. 2019Mean-field Markov decision processes with common noise and open-loop controlscited by 25
  7. 2023Recent Developments in Machine Learning Methods for Stochastic Control and Gamescited by 24
  8. 2023Actor-Critic learning for mean-field control in continuous timecited by 22
  9. 2023Continuous-time q-learning for mean-field control problemscited by 22
  10. 2023Efficient Exploration in Continuous-time Model-based Reinforcement Learningcited by 22
  11. 2021Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex modelscited by 21
  12. 2019Stochastic Graphon Games: I. The Static Casecited by 13
6 more
  1. 2024Unified continuous-time q-learning for mean-field game and mean-field control problemscited by 13
  2. 2021Finite State Graphon Games with Applications to Epidemicscited by 10
  3. 2022Optimal scheduling of entropy regulariser for continuous-time linear-quadratic reinforcement learningcited by 6
  4. 2022Temporal Difference Learning with Continuous Time and State in the Stochastic Settingcited by 5
  5. 2022A Non-asymptotic Analysis of Non-parametric Temporal-Difference Learningcited by 4
  6. 2023Policy Iteration Reinforcement Learning Method for Continuous-Time Linear-Quadratic Mean-Field Control Problemscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundationscited by 6
  2. 2026Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Datacited by 1
  3. 2026Mean Field Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mathematical methods of reinforcement learning
  2. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  3. 2026Discretization error from regularized Reinforcement Learning to continuous-time stochastic control
  4. 2026Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence
  5. 2025Reinforcement Learning with Random Time Horizons
  6. 2025Control and Reinforcement Learning through the Lens of Optimization: An Algorithmic Perspective
  7. 2024When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL
  8. 2024Control randomisation approach for policy gradient and application to reinforcement learning in optimal switching