mlsanityData to 7 Oct 2026

Exploratory HJB equations and their convergence

arXiv 2109.10269 ↗First posted 21 Sep 2021math.OC math.AP math.PRcited by 28

Not on the map. Why?

Abstract (arXiv)

We study the exploratory Hamilton--Jacobi--Bellman (HJB) equation arising from the entropy-regularized exploratory control problem, which was formulated by Wang, Zariphopoulou and Zhou (J. Mach. Learn. Res., 21, 2020) in the context of reinforcement learning in continuous time and space. We establish the well-posedness and regularity of the viscosity solution to the equation, as well as the convergence of the exploratory control problem to the classical stochastic control problem when the level of exploration decays to zero. We then apply the general results to the exploratory temperature control problem, which was introduced by Gao, Xu and Zhou (arXiv:2005.04057, 2020) to design an endogenous temperature schedule for simulated annealing (SA) in the context of non-convex optimization. We derive an explicit rate of convergence for this problem as exploration diminishes to zero, and find that the steady state of the optimally controlled process exists, which is however neither a Dirac mass on the global optimum nor a Gibbs measure.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Sep 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 3; 2024: 8; 2025: 8; 2026: 5.

28 papers.

Sort:

By area:

  1. 2021Recent Advances in Reinforcement Learning in Financecited by 102
  2. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
  3. 2022q-Learning in Continuous Timecited by 68
  4. 2024Fine-tuning of diffusion models via stochastic control: entropy regularization and beyondcited by 42
  5. 2023Policy Optimization for Continuous Reinforcement Learningcited by 30
  6. 2024Reward-Directed Score-Based Diffusion Models via q-Learningcited by 18
  7. 2023Fast Policy Learning for Linear Quadratic Control with Entropy Regularizationcited by 11
  8. 2024Regret of exploratory policy improvement and q-learningcited by 10
  9. 2024Entropy annealing for policy mirror descent in continuous time and spacecited by 9
  10. 2023A Policy Gradient Framework for Stochastic Optimal Control Problems with Global Convergence Guaranteecited by 6
  11. 2024Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penaltycited by 5
  12. 2024Exploratory Optimal Stopping: A Singular Control Formulationcited by 5
16 more
  1. 2024Convergence Analysis for Entropy-Regularized Control Problems: A Probabilistic Approachcited by 4
  2. 2025Continuous-time reinforcement learning for optimal switching over multiple regimescited by 4
  3. 2025A Temporal Difference Method for Stochastic Continuous Dynamicscited by 3
  4. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Controlcited by 3
  5. 2026Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problemscited by 3
  6. 2022Choquet regularization for reinforcement learningcited by 2
  7. 2024A random measure approach to reinforcement learning in continuous timecited by 1
  8. 2025Exploratory Utility Maximization Problem with Tsallis Entropycited by 1
  9. 2025Continuous Policy and Value Iteration for Relaxed Stochastic Control Problems and Its Convergencecited by 1
  10. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Modelscited by 1
  11. 2025A Robust Model-Based Approach for Continuous-Time Policy Evaluation with Unknown L'evy Process Dynamicscited by 0
  12. 2025Robust Exploratory Stopping under Ambiguity in Reinforcement Learningcited by 0
  13. 2025Risk-Sensitive Q-Learning in Continuous Time with Application to Dynamic Portfolio Selectioncited by 0
  14. 2026Reinforcement Learning for Speculative Trading under Exploratory Frameworkcited by 0
  15. 2026Entropy-Regularized Reinforcement Learning for Linear-Quadratic Stackelberg Differential Games in Regime-Switching Diffusion Modelscited by 0
  16. 2026Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergencecited by 0