# Approximate Q-Learning for Controlled Diffusion Processes and its Near Optimality

arXiv: https://arxiv.org/abs/2203.07499 · HTML: https://mlsanity.com/paper/2203.07499/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2022.

## Built on by (most cited first, 7 of 7)

- [Q-Learning for Stochastic Control under General Information Structures and Non-Markovian Environments](https://mlsanity.com/paper/2311.00123/) (2023)
- [Discretization error from regularized Reinforcement Learning to continuous-time stochastic control](https://mlsanity.com/paper/2604.21179/) (2026)
- [Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-off](https://mlsanity.com/paper/2212.08949/) (2022)
- [Physics-informed RL for Maximal Safety Probability Estimation](https://mlsanity.com/paper/2403.16391/) (2024)
- [Reinforcement Learning in High-frequency Market Making](https://mlsanity.com/paper/2407.21025/) (2024)
- [Two Kinds of Learning Algorithms for Continuous-Time VWAP Targeting Execution](https://mlsanity.com/paper/2411.06645/) (2024)
- [Reinforcement Learning for Discounted and Ergodic Control of Diffusion Processes](https://mlsanity.com/paper/2603.13155/) (2026)

## Abstract (arXiv)

We study a Q learning algorithm for continuous time stochastic control problems. The proposed algorithm uses the sampled state process by discretizing the state and control action spaces under piece-wise constant control processes. We show that the algorithm converges to the optimality equation of a finite Markov decision process (MDP). Using this MDP model, we provide an upper bound for the approximation error for the optimal value function of the continuous time control problem. Furthermore, we present provable upper-bounds for the performance loss of the learned control process compared to the optimal admissible control process of the original problem. The provided error upper-bounds are functions of the time and space discretization parameters, and they reveal the effect of different levels of the approximation: (i) approximation of the continuous time control problem by an MDP, (ii) use of piece-wise constant control processes, (iii) space discretization. Finally, we state a time complexity bound for the proposed algorithm as a function of the time and space discretization parameters.