# Regret of exploratory policy improvement and q-learning

arXiv: https://arxiv.org/abs/2411.01302 · HTML: https://mlsanity.com/paper/2411.01302/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 53 references that are in the map.
- First posted on arXiv 2 Nov 2024.

## Builds on (most cited first, 17 of 17)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Learning to Navigate in Complex Environments](https://mlsanity.com/paper/1611.03673/) (2016)
- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [q-Learning in Continuous Time](https://mlsanity.com/paper/2207.00713/) (2022)
- [Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach](https://mlsanity.com/paper/2108.06655/) (2021)
- [Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorial](https://mlsanity.com/paper/2402.07487/) (2024)
- [Exploratory HJB equations and their convergence](https://mlsanity.com/paper/2109.10269/) (2021)
- [Policy Optimization for Continuous Reinforcement Learning](https://mlsanity.com/paper/2305.18901/) (2023)
- [Reward-Directed Score-Based Diffusion Models via q-Learning](https://mlsanity.com/paper/2409.04832/) (2024)
- [Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning](https://mlsanity.com/paper/2409.08400/) (2024)
- [Entropy annealing for policy mirror descent in continuous time and space](https://mlsanity.com/paper/2405.20250/) (2024)
- [Policy Iteration for Exploratory Hamilton--Jacobi--Bellman Equations](https://mlsanity.com/paper/2406.00612/) (2024)
- [Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems](https://mlsanity.com/paper/2407.17226/) (2024)
- [Policy iteration for the deterministic control problems -- a viscosity approach](https://mlsanity.com/paper/2301.00419/) (2023)
- [Convergence Analysis for Entropy-Regularized Control Problems: A Probabilistic Approach](https://mlsanity.com/paper/2406.10959/) (2024)

## Built on by (most cited first, 12 of 12)

- [Conditional Diffusion Guidance under Hard Constraint: A Stochastic Analysis Approach](https://mlsanity.com/paper/2602.05533/) (2026)
- [Continuous-time reinforcement learning for optimal switching over multiple regimes](https://mlsanity.com/paper/2512.04697/) (2025)
- [ART for Diffusion Sampling: A Reinforcement Learning Approach to Timestep Schedule](https://mlsanity.com/paper/2601.18681/) (2026)
- [Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems](https://mlsanity.com/paper/2507.00358/) (2025)
- [A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models](https://mlsanity.com/paper/2607.14522/) (2026)
- [Continuous Policy and Value Iteration for Relaxed Stochastic Control Problems and Its Convergence](https://mlsanity.com/paper/2506.08121/) (2025)
- [Amortized Guidance for Image Inpainting with Pretrained Diffusion Models](https://mlsanity.com/paper/2605.13010/) (2026)
- [Continuous-time reinforcement learning: ellipticity enables model-free value function approximation](https://mlsanity.com/paper/2602.06930/) (2026)
- [ART for Diffusion Sampling: Continuous-Time Control and Actor-Critic Learning](https://mlsanity.com/paper/2607.02137/) (2026)
- [Continuous-Time Reinforcement Learning for N-Player Stochastic Differential Games with Exploratory Policies](https://mlsanity.com/paper/2607.19928/) (2026)
- [Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence](https://mlsanity.com/paper/2609.14972/) (2026)
- [Amortized Score-Hamiltonian Policy Iteration: A Grid-Free Scheme for Relaxed Stochastic Control Problems](https://mlsanity.com/paper/2610.04285/) (2026)

## Abstract (arXiv)

We study the convergence of q-learning and related algorithms introduced by Jia and Zhou (J. Mach. Learn. Res., 24 (2023), 161) for controlled diffusion processes. For exploratory policy improvement, we establish exponential convergence under growth and regularity assumptions on the model parameters. For q-learning, we derive quantitative error and regret bounds under additional assumptions on the function approximation and the associated stochastic approximation dynamics.