mlsanityData to 14 Oct 2026

Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimation

arXiv 2412.11138 ↗First posted 15 Dec 2024cs.LG cs.AIcited by 3

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

A key aspect of Safe Reinforcement Learning (Safe RL) involves estimating the constraint condition for the next policy, which is crucial for guiding the optimization of safe policy updates. However, the existing Advantage-based Estimation (ABE) method relies on the infinite-horizon discounted advantage function. This dependence leads to catastrophic errors in finite-horizon scenarios with non-discounted constraints, resulting in safety-violation updates. In response, we propose the first estimation method for finite-horizon non-discounted constraints in deep Safe RL, termed Gradient-based Estimation (GBE), which relies on the analytic gradient derived along trajectories. Our theoretical and empirical analyses demonstrate that GBE can effectively estimate constraint changes over a finite horizon. Constructing a surrogate optimization problem with GBE, we developed a novel Safe RL algorithm called Constrained Gradient-based Policy Optimization (CGPO). CGPO identifies feasible optimal policies by iteratively resolving sub-problems within trust regions. Our empirical results reveal that CGPO, unlike baseline algorithms, successfully estimates the constraint functions of subsequent policies, thereby ensuring the efficiency and feasibility of each update.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 29 of its 60 references that are in the map. First posted on arXiv 15 Dec 2024.

Builds on

29 of 60 references.

References · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  5. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  6. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  7. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  8. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  9. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  10. 2023AI Alignment: A Comprehensive Surveycited by 263
  11. 2020Projection-Based Constrained Policy Optimizationcited by 203
  12. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
17 more
  1. 2016A Differentiable Physics Engine for Deep Learning in Roboticscited by 108
  2. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 104
  3. 2022Do Differentiable Simulators Give Better Policy Gradients?cited by 89
  4. 2021Gradients are Not All You Needcited by 86
  5. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  6. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 71
  7. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 63
  8. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 58
  9. 2022Constrained Policy Optimization via Bayesian World Modelscited by 57
  10. 2023FluidLab: A Differentiable Environment for Benchmarking Complex Fluid Manipulationcited by 40
  11. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 36
  12. 2020A Sample-Efficient Algorithm for Episodic Finite-Horizon MDP with Constraintscited by 34
  13. 2022Dojo: A Differentiable Physics Engine for Roboticscited by 26
  14. 2018Reparameterization Gradient for Non-differentiable Modelscited by 11
  15. 2022A policy gradient approach for Finite Horizon Constrained Markov Decision Processescited by 8
  16. 2018A Unified Approach for Multi-step Temporal-Difference Learning with Eligibility Traces in Reinforcement Learningcited by 4
  17. 2019TBQ(σ): Improving Efficiency of Trace Utilization for Off-Policy Reinforcement Learningcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularizationcited by 14
  2. 2025Consensus Based Stochastic Controlcited by 0
  3. 2025Meta-Reinforcement Learning for Fast and Data-Efficient Spectrum Allocation in Dynamic Wireless Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Anytime Safe Reinforcement Learning
  2. 2025Exchange Policy Optimization Algorithm for Semi-Infinite Safe Reinforcement Learning
  3. 2025Feasibility-Aware Pessimistic Estimation: Toward Long-Horizon Safety in Offline RL
  4. 2025Off-Policy Reinforcement Learning with Anytime Safety Guarantees via Robust Safe Gradient Flow
  5. 2026CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning
  6. 2024A Survey of Constraint Formulations in Safe Reinforcement Learning
  7. 2024Off-Policy Primal-Dual Safe Reinforcement Learning
  8. 2026Off-Policy Safe Reinforcement Learning with Constrained Optimistic Exploration