# Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Time

arXiv: https://arxiv.org/abs/1912.10697 · HTML: https://mlsanity.com/paper/1912.10697/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2019.

## Built on by (most cited first, 9 of 9)

- [Continuous-Time Fitted Value Iteration for Robust Policies](https://mlsanity.com/paper/2110.01954/) (2021)
- [Universal Approximation Theorem of Deep Q-Networks](https://mlsanity.com/paper/2505.02288/) (2025)
- [Offline Reinforcement Learning at Multiple Frequencies](https://mlsanity.com/paper/2207.13082/) (2022)
- [A Holistic Power Optimization Approach for Microgrid Control Based on Deep Reinforcement Learning](https://mlsanity.com/paper/2403.01013/) (2024)
- [Optimisation of Structured Neural Controller Based on Continuous-Time Policy Gradient](https://mlsanity.com/paper/2201.06262/) (2022)
- [GNN-Assisted Phase Space Integration with Application to Atomistics](https://mlsanity.com/paper/2303.16088/) (2023)
- [Why does the two-timescale Q-learning converge to different mean field solutions? A unified convergence analysis](https://mlsanity.com/paper/2404.04357/) (2024)
- [Optimal Control of Fluid Restless Multi-armed Bandits: A Machine Learning Approach](https://mlsanity.com/paper/2502.03725/) (2025)
- [PhiBE-Q-Learning: Bridging Off-Policy Reinforcement Learning and Continuous-Time Control](https://mlsanity.com/paper/2606.21925/) (2026)

## Abstract (arXiv)

In this paper, we introduce Hamilton-Jacobi-Bellman (HJB) equations for Q-functions in continuous time optimal control problems with Lipschitz continuous controls. The standard Q-function used in reinforcement learning is shown to be the unique viscosity solution of the HJB equation. A necessary and sufficient condition for optimality is provided using the viscosity solution framework. By using the HJB equation, we develop a Q-learning method for continuous-time dynamical systems. A DQN-like algorithm is also proposed for high-dimensional state and control spaces. The performance of the proposed Q-learning algorithm is demonstrated using 1-, 10- and 20-dimensional dynamical systems.