# Discretization error from regularized Reinforcement Learning to continuous-time stochastic control

arXiv: https://arxiv.org/abs/2604.21179 · HTML: https://mlsanity.com/paper/2604.21179/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 16 of its 47 references that are in the map.
- First posted on arXiv 23 Apr 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 16 of 16)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [Lipschitz Continuity in Model-based Reinforcement Learning](https://mlsanity.com/paper/1804.07193/) (2018)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Exploratory HJB equations and their convergence](https://mlsanity.com/paper/2109.10269/) (2021)
- [Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems](https://mlsanity.com/paper/2211.00617/) (2022)
- [Reinforcement learning for linear-convex models with jumps via stability analysis of feedback controls](https://mlsanity.com/paper/2104.09311/) (2021)
- [State-Dependent Temperature Control for Langevin Diffusions](https://mlsanity.com/paper/2011.07456/) (2020)
- [Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning](https://mlsanity.com/paper/2503.09981/) (2025)
- [Convergence of Policy Iteration for Entropy-Regularized Stochastic Control Problems](https://mlsanity.com/paper/2209.07059/) (2022)
- [Approximate Q-Learning for Controlled Diffusion Processes and its Near Optimality](https://mlsanity.com/paper/2203.07499/) (2022)
- [On Bellman equations for continuous-time policy evaluation I: discretization and approximation](https://mlsanity.com/paper/2407.05966/) (2024)
- [PhiBE: A PDE-based Bellman Equation for Continuous Time Policy Evaluation](https://mlsanity.com/paper/2405.12535/) (2024)
- [Optimal-PhiBE: A PDE-based Model-free framework for Continuous-time Reinforcement Learning](https://mlsanity.com/paper/2506.05208/) (2025)
- [On the grid-sampling limit SDE](https://mlsanity.com/paper/2410.07778/) (2024)

## Built on by (most cited first, 2 of 2)

- [Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data](https://mlsanity.com/paper/2606.26498/) (2026)
- [Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence](https://mlsanity.com/paper/2609.14972/) (2026)

## Abstract (arXiv)

This paper establishes a rigorous connection between regularized discrete-time reinforcement learning (RL) and continuous-time stochastic optimal control. Specifically, classical RL algorithms are typically solving a regularized discrete-time Bellman equation. We study the discretization error, namely, the gap between the optimal policy induced by the regularized discrete-time Bellman equation and the true optimal feedback control of the underlying continuous-time stochastic control problem. By deriving quantitative convergence rates for this gap, we provide a rigorous foundation for understanding the stability and implementation of exploratory RL policies in stochastic continuous-time environments.