mlsanityData to 7 Oct 2026

Regret of exploratory policy improvement and q-learning

arXiv 2411.01302 ↗First posted 2 Nov 2024cs.LG math.OC math.PRcited by 10

Not on the map. Why?

Abstract (arXiv)

We study the convergence of q-learning and related algorithms introduced by Jia and Zhou (J. Mach. Learn. Res., 24 (2023), 161) for controlled diffusion processes. For exploratory policy improvement, we establish exponential convergence under growth and regularity assumptions on the model parameters. For q-learning, we derive quantitative error and regret bounds under additional assumptions on the function approximation and the associated stochastic approximation dynamics.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 17 of its 53 references that are in the map. First posted on arXiv 2 Nov 2024.

Builds on

17 of 53 references.

References · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  3. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  4. 2016Learning to Navigate in Complex Environmentscited by 192
  5. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
  6. 2022q-Learning in Continuous Timecited by 68
  7. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 62
  8. 2024Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorialcited by 60
  9. 2023Policy Optimization for Continuous Reinforcement Learningcited by 30
  10. 2021Exploratory HJB equations and their convergencecited by 28
  11. 2024Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learningcited by 19
  12. 2024Reward-Directed Score-Based Diffusion Models via q-Learningcited by 18
5 more
  1. 2024Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problemscited by 10
  2. 2024Entropy annealing for policy mirror descent in continuous time and spacecited by 9
  3. 2024Policy Iteration for Exploratory Hamilton--Jacobi--Bellman Equationscited by 9
  4. 2023Policy iteration for the deterministic control problems -- a viscosity approachcited by 7
  5. 2024Convergence Analysis for Entropy-Regularized Control Problems: A Probabilistic Approachcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2026Conditional Diffusion Guidance under Hard Constraint: A Stochastic Analysis Approachcited by 10
  2. 2025Continuous-time reinforcement learning for optimal switching over multiple regimescited by 4
  3. 2026ART for Diffusion Sampling: A Reinforcement Learning Approach to Timestep Schedulecited by 3
  4. 2025Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problemscited by 2
  5. 2025Continuous Policy and Value Iteration for Relaxed Stochastic Control Problems and Its Convergencecited by 1
  6. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Modelscited by 1
  7. 2026Continuous-time reinforcement learning: ellipticity enables model-free value function approximationcited by 0
  8. 2026Amortized Guidance for Image Inpainting with Pretrained Diffusion Modelscited by 0
  9. 2026ART for Diffusion Sampling: Continuous-Time Control and Actor-Critic Learningcited by 0
  10. 2026Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergencecited by 0