Reinforcement Learning for Discounted and Ergodic Control of Diffusion Processes
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
This paper develops a quantized Q-learning algorithm for the optimal control of controlled diffusion processes on ℝd under both discounted and ergodic (average) cost criteria. We first establish near-optimality of finite-state MDP approximations to discrete-time discretizations of the diffusion, then introduce a quantized Q-learning scheme and prove its almost-sure convergence to near-optimal policies for the finite MDP. These policies, when interpolated to continuous time, are shown to be near-optimal for the original diffusion model under discounted costs and -- via a vanishing-discount argument -- also under ergodic costs for sufficiently small discount factors. The analysis applies under mild conditions (Lipschitz dynamics, non-degeneracy, bounded continuous costs, and Lyapunov stability for ergodic case) without requiring prior knowledge of the system dynamics or restrictions on control policies (beyond admissibility). Our results complement recent work on continuous-time reinforcement learning for diffusions by providing explicit near-optimality rates and extending rigorous guarantees both for discounted cost and ergodic cost criteria for diffusions with unbounded state space.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 38 references that are in the map. First posted on arXiv 13 Mar 2026.
Builds on
12 of 38 references.
Closest ancestors · 1
Same direction, newest first
Other references · 11
- 2022q-Learning in Continuous Timecited by 91
- 2015Asymptotic Optimality of Finite Approximations to Markov Decision Processes with Borel Spacescited by 14
- 2021Q-Learning for MDPs with General Spaces: Convergence and Near Optimality via Quantization under Weak Continuitycited by 11
- 2022Approximate Q-Learning for Controlled Diffusion Processes and its Near Optimalitycited by 7
- 2022Continuity of Cost in Borkar Control Topology and Implications on Discrete Space and Time Approximations for Controlled Diffusions under Several Criteriacited by 3
- 2023Approximations and Learning for Continuous State and Action MDPs under Average Cost Criteriacited by 3
- 2023Reinforcement Learning for Near-Optimal Design of Zero-Delay Codes for Markov Sourcescited by 3
- 2025Near Optimality of Discrete-Time Approximations for Controlled McKean-Vlasov and Large Interacting Particle Diffusionscited by 2
- 2015Ergodicity of robust switching control and nonlinear system of quasi variational inequalitiescited by 1
- 2025Discrete-Time Approximations of Controlled Diffusions with Infinite Horizon Discounted and Average Costcited by 1
- 2025Adaptive Partitioning and Learning for Stochastic Control of Diffusion Processescited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Continuous-time q-Learning for Jump-Diffusion Models under Tsallis Entropy
- 2026Reinforcement Learning with Function Approximation for Non-Markov Processes
- 2023Q-Learning for Stochastic Control under General Information Structures and Non-Markovian Environments
- 2026Reinforcement Learning for Exponential Utility: Algorithms and Convergence in Discounted MDPs
- 2024Regret of exploratory policy improvement and q-learning
- 2024On Bellman equations for continuous-time policy evaluation I: discretization and approximation
- 2026Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence
- 2026Reinforcement Learning for Continuous-Time Jump Markov Decision Processes with Applications to Network Dynamic Pricing