mlsanityData to 14 Oct 2026

Reachability-Preserving Bellman Operator for the Discounted Reach-Cost Value Function: Uniting Hamilton-Jacobi Reachability and Reinforcement Learning

arXiv 2607.07893 ↗First posted 8 Jul 2026eess.SY cs.SYcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Hamilton-Jacobi (HJ) reachability provides rigorous safety and reachability guarantees for continuous-time dynamical systems, but its numerical solution suffers from the curse of dimensionality. Deep reinforcement learning (DRL), by contrast, offers scalable sample-based methods. However, RL is typically built around additive cumulative rewards; whereas, reachability objectives are inherently non-additive. This mismatch makes a direct bridge between HJ reachability and RL nontrivial. Recent discounted formulations have either introduced contraction by altering the original reachability semantics, or preserved exact semantics on the HJ side without a corresponding Bellman fixed-point characterization. In this paper, we close this gap by building on a semantics-preserving discounted reach-based value function and deriving a non-additive Bellman operator whose unique fixed point exactly matches the value function in the HJ formulation. We prove that discounting makes this operator contractive, yielding existence, uniqueness, and convergence of value iteration. Furthermore, we establish the equivalence between the HJ and Bellman characterizations, and show that RL can be interpreted as a sample-based approximation scheme for the same fixed-point equation. This yields a principled and semantically exact connection between HJ reachability and RL, enabling learning-based methods to approximate reachability value functions while preserving their safety-critical meaning. As a result, the proposed framework opens the door to scalable, data-driven computation of reachable sets and safety certificates in high-dimensional systems. Numerical experiments demonstrate close agreement with HJ solutions, confirm preservation of reachability semantics via alignment of zero level sets, and support the interpretation of reinforcement learning as a sample-based solver of the proposed Bellman operator.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 27 references that are in the map. First posted on arXiv 8 Jul 2026.

Builds on

13 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Certifying Hamilton-Jacobi Reachability Learned via Reinforcement Learning18 Feb
  2. 2025Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulations19 Jun

Other references · 11

  1. 2017Hamilton-Jacobi Reachability: A Brief Overview and Recent Advancescited by 201
  2. 2020DeepReach: A Deep Learning Approach to High-Dimensional Reachabilitycited by 91
  3. 2021Robust Control Barrier-Value Functions for Safety-Critical Controlcited by 51
  4. 2021Safety and Liveness Guarantees through Reach-Avoid Reinforcement Learningcited by 49
  5. 2016Algorithms for Overcoming the Curse of Dimensionality for Certain Hamilton-Jacobi Equations Arising in Control Theory and Elsewherecited by 24
  6. 2024Hamilton-Jacobi Reachability in Reinforcement Learning: A Surveycited by 16
  7. 2021Scalable Learning of Safety Guarantees for Autonomous Systems using Hamilton-Jacobi Reachabilitycited by 14
  8. 2018A Minimum Discounted Reward Hamilton-Jacobi Formulation for Computing Reachable Setscited by 11
  9. 2018Inner-Approximating Reachable Sets for Polynomial Systems with Time-Varying Uncertaintiescited by 4
  10. 2026Bellman Value Decomposition for Task Logic in Safe Optimal Controlcited by 4
  11. 2024Converse Barrier Certificates for Finite-time Safety Verification of Continuous-time Perturbed Deterministic Systemscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Unifying Hamilton-Jacobi Reachability and Reinforcement Learning
  2. 2025Theoretical Barriers in Bellman-Based Reinforcement Learning
  3. 2026AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learning
  4. 2025A Temporal Difference Method for Stochastic Continuous Dynamics
  5. 2025Bellman operator convergence enhancements in reinforcement learning algorithms
  6. 2025Is Bellman Equation Enough for Learning Control?
  7. 2026Deep Q-Learning on H"older Spaces
  8. 2025Continuous-Time Value Iteration for Multi-Agent Reinforcement Learning