# Reinforcement Learning for Discounted and Ergodic Control of Diffusion Processes

arXiv: https://arxiv.org/abs/2603.13155 · HTML: https://mlsanity.com/paper/2603.13155/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 12 of its 38 references that are in the map.
- First posted on arXiv 13 Mar 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 12 of 12)

- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Asymptotic Optimality of Finite Approximations to Markov Decision Processes with Borel Spaces](https://mlsanity.com/paper/1503.02244/) (2015)
- [Q-Learning for MDPs with General Spaces: Convergence and Near Optimality via Quantization under Weak Continuity](https://mlsanity.com/paper/2111.06781/) (2021)
- [Approximate Q-Learning for Controlled Diffusion Processes and its Near Optimality](https://mlsanity.com/paper/2203.07499/) (2022)
- [Continuity of Cost in Borkar Control Topology and Implications on Discrete Space and Time Approximations for Controlled Diffusions under Several Criteria](https://mlsanity.com/paper/2209.14982/) (2022)
- [Approximations and Learning for Continuous State and Action MDPs under Average Cost Criteria](https://mlsanity.com/paper/2308.07591/) (2023)
- [Reinforcement Learning for Near-Optimal Design of Zero-Delay Codes for Markov Sources](https://mlsanity.com/paper/2311.12609/) (2023)
- [Quantizer Design for Finite Model Approximations, Model Learning, and Quantized Q-Learning for MDPs with Unbounded Spaces](https://mlsanity.com/paper/2510.04355/) (2025)
- [Near Optimality of Discrete-Time Approximations for Controlled McKean-Vlasov and Large Interacting Particle Diffusions](https://mlsanity.com/paper/2510.21208/) (2025)
- [Ergodicity of robust switching control and nonlinear system of quasi variational inequalities](https://mlsanity.com/paper/1501.04477/) (2015)
- [Discrete-Time Approximations of Controlled Diffusions with Infinite Horizon Discounted and Average Cost](https://mlsanity.com/paper/2502.05596/) (2025)
- [Adaptive Partitioning and Learning for Stochastic Control of Diffusion Processes](https://mlsanity.com/paper/2512.14991/) (2025)

## Abstract (arXiv)

This paper develops a quantized Q-learning algorithm for the optimal control of controlled diffusion processes on ℝd under both discounted and ergodic (average) cost criteria. We first establish near-optimality of finite-state MDP approximations to discrete-time discretizations of the diffusion, then introduce a quantized Q-learning scheme and prove its almost-sure convergence to near-optimal policies for the finite MDP. These policies, when interpolated to continuous time, are shown to be near-optimal for the original diffusion model under discounted costs and -- via a vanishing-discount argument -- also under ergodic costs for sufficiently small discount factors. The analysis applies under mild conditions (Lipschitz dynamics, non-degeneracy, bounded continuous costs, and Lyapunov stability for ergodic case) without requiring prior knowledge of the system dynamics or restrictions on control policies (beyond admissibility). Our results complement recent work on continuous-time reinforcement learning for diffusions by providing explicit near-optimality rates and extending rigorous guarantees both for discounted cost and ergodic cost criteria for diffusions with unbounded state space.