mlsanityData to 14 Oct 2026

Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learning

arXiv 2205.12184 ↗First posted 24 May 2022cs.LG math.OC stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

Continuous-time reinforcement learning offers an appealing formalism for describing control problems in which the passage of time is not naturally divided into discrete increments. Here we consider the problem of predicting the distribution of returns obtained by an agent interacting in a continuous-time, stochastic environment. Accurate return predictions have proven useful for determining optimal policies for risk-sensitive control, learning state representations, multiagent coordination, and more. We begin by establishing the distributional analogue of the Hamilton-Jacobi-Bellman (HJB) equation for Ito diffusions and the broader class of Feller-Dynkin processes. We then specialize this equation to the setting in which the return distribution is approximated by N uniformly-weighted particles, a common design choice in distributional algorithms. Our derivation highlights additional terms due to statistical diffusivity which arise from the proper handling of distributions in the continuous-time setting. Based on this, we propose a tractable algorithm for approximately solving the distributional HJB based on a JKO scheme, which can be implemented in an online control algorithm. We demonstrate the effectiveness of such an algorithm in a synthetic control problem.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 17 of its 45 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

17 of 45 references.

References · 17

  1. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  2. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  3. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  4. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  5. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  6. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  7. 2013Risk-sensitive Reinforcement Learningcited by 130
  8. 2014Optimizing the CVaR via Samplingcited by 99
  9. 2014Algorithms for CVaR Optimization in MDPscited by 99
  10. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
  11. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 63
  12. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 60
5 more
  1. 2019Making Deep Q-learning methods robust to time discretizationcited by 58
  2. 2015Increasing the Action Gap: New Operators for Reinforcement Learningcited by 49
  3. 2018Policy Optimization as Wasserstein Gradient Flowscited by 34
  4. 2019Stochastically Dominant Distributional Reinforcement Learningcited by 12
  5. 2021Distributional Offline Continuous-Time Reinforcement Learning with Neural Physics-Informed PDEs (SciPhy RL for DOCTR-L)cited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 1; 2025: 5; 2026: 3.

12 papers.

  1. 2023Bridging Physics-Informed Neural Networks with Reinforcement Learning: Hamilton-Jacobi-Bellman Proximal Policy Optimization (HJBPPO)cited by 7
  2. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learningcited by 7
  3. 2025A Temporal Difference Method for Stochastic Continuous Dynamicscited by 3
  4. 2022Decision-making with Speculative Opponent Modelscited by 2
  5. 2023Actor-Critic Methods using Physics-Informed Neural Networks: Control of a 1D PDE Model for Fluid-Cooled Battery Packscited by 2
  6. 2025Parabolic Continual Learningcited by 2
  7. 2025Tractable Representations for Convergent Approximation of Distributional HJB Equationscited by 2
  8. 2025Continuous-Time Value Iteration for Multi-Agent Reinforcement Learningcited by 2
  9. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learningcited by 2
  10. 2026Certifying Hamilton-Jacobi Reachability Learned via Reinforcement Learningcited by 1
  11. 2026Unifying Hamilton-Jacobi Reachability and Reinforcement Learningcited by 0
  12. 2026Markov Decision Processes of the Third Kind: Learning Distributions by Policy Gradient Descentcited by 0