Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulations

arXiv 2506.16016 ↗First posted 19 Jun 2025cs.AI cs.SY eess.SYcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Hard constraints in reinforcement learning (RL) often degrade policy performance. Lagrangian methods offer a way to blend objectives with constraints, but require intricate reward engineering and parameter tuning. In this work, we extend recent advances that connect Hamilton-Jacobi (HJ) equations with RL to propose two novel value functions for dual-objective satisfaction. Namely, we address: 1) the Reach-Always-Avoid (RAA) problem -- of achieving distinct reward and penalty thresholds -- and 2) the Reach-Reach (RR) problem -- of achieving thresholds of two distinct rewards. In contrast with temporal logic approaches, which typically involve representing an automaton, we derive explicit, tractable Bellman forms in this context via decomposition. Specifically, we prove that the RAA and RR problems may be rewritten as compositions of previously studied HJ-RL problems. We leverage our analysis to propose a variation of Proximal Policy Optimization (DOHJ-PPO), and demonstrate that it produces distinct behaviors from previous approaches, outcompeting a number of baselines in success, safety and speed across a range of tasks for safe-arrival and multi-target achievement.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 36 of its 58 references that are in the map. First posted on arXiv 19 Jun 2025.

Builds on

36 of 58 references.

Closest ancestors · 1

Same direction, newest first

2024Solving Minimum-Cost Reach Avoid using Reinforcement Learning29 Oct

Other references · 35

2017Proximal Policy Optimization Algorithmscited by 15,215
2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
2017Constrained Policy Optimizationcited by 983
2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
2016Control Barrier Function Based Quadratic Programs for Safety Critical Systemscited by 287
2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
2020Projection-Based Constrained Policy Optimizationcited by 203
2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 177
2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 158
2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134

23 more

2016Reinforcement Learning With Temporal Logic Rewardscited by 117
2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 115
2018Dynamic Weights in Multi-Objective Deep Reinforcement Learningcited by 100
2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 88
2016Multi-Objective Deep Reinforcement Learningcited by 83
2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 78
2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 66
2022Reachability Constrained Reinforcement Learningcited by 59
2017Hybrid Reward Architecture for Reinforcement Learningcited by 58
2019Exploration via Hindsight Goal Generationcited by 54
2014A Learning Based Approach to Control Synthesis of Markov Decision Processes for Linear Temporal Logic Specificationscited by 49
2021Safety and Liveness Guarantees through Reach-Avoid Reinforcement Learningcited by 49
2014Reach-Avoid Problems with Time-Varying Dynamics, Targets and Constraintscited by 44
2021A Primal-Dual Approach to Constrained Markov Decision Processescited by 42
2014Verification of Markov Decision Processes using Learning Algorithmscited by 39
2021A Simple Reward-free Approach to Constrained Reinforcement Learningcited by 30
2023Solving Stabilize-Avoid Optimal Control via Epigraph Form and Deep Reinforcement Learningcited by 24
2021Faster Algorithm and Sharper Analysis for Constrained Markov Decision Processcited by 19
2022Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificatecited by 18
2024Constrained Reinforcement Learning with Smoothed Log Barrier Functioncited by 15
2023Learning Stabilization Control from Observations by Learning Lyapunov-like Proxy Modelscited by 10
2023Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewardscited by 8
2023Reinforcement Learning with Non-Cumulative Objectivecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way