mlsanityData to 14 Oct 2026

Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning

arXiv 2410.05655 ↗First posted 8 Oct 2024cs.LGcited by 6

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

In reinforcement learning, classic on-policy evaluation methods often suffer from high variance and require massive online data to attain the desired accuracy. Previous studies attempt to reduce evaluation variance by searching for or designing proper behavior policies to collect data. However, these approaches ignore the safety of such behavior policies -- the designed behavior policies have no safety guarantee and may lead to severe damage during online executions. In this paper, to address the challenge of reducing variance while ensuring safety simultaneously, we propose an optimal variance-minimizing behavior policy under safety constraints. Theoretically, while ensuring safety constraints, our evaluation method is unbiased and has lower variance than on-policy evaluation. Empirically, our method is the only existing method to achieve both substantial variance reduction and safety constraint satisfaction. Furthermore, we show our method is even superior to previous methods in both variance reduction and execution safety.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 26 of its 56 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

26 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024A Survey of Constraint Formulations in Safe Reinforcement Learning3 Feb

Other references · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021Convex optimizationcited by 2,515
  4. 2016OpenAI Gymcited by 1,736
  5. 2017Constrained Policy Optimizationcited by 983
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  8. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  9. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  10. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  11. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  12. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
13 more
  1. 2019Batch Policy Learning under Constraintscited by 244
  2. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  3. 2017A Rewriting System for Convex Optimization Problemscited by 179
  4. 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 115
  5. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  6. 2017Data-Efficient Policy Evaluation Through Behavior Policy Searchcited by 26
  7. 2024The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noisecited by 24
  8. 2022ReVar: Strengthening Policy Evaluation via Reduced Variance Samplingcited by 16
  9. 2021Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learningcited by 15
  10. 2023Efficient Policy Evaluation with Offline Data Informed Behavior Policy Designcited by 14
  11. 2022Semi-analytical Industrial Cooling System Model for Reinforcement Learningcited by 9
  12. 2024Efficient Multi-Policy Evaluation for Reinforcement Learningcited by 9
  13. 2024Doubly Optimal Policy Evaluation for Reinforcement Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Offline Two-Player Zero-Sum Markov Games with KL Regularizationcited by 6
  2. 2026Pessimism-Free Offline Learning in General-Sum Games via KL Regularizationcited by 5
  3. 2026Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluationcited by 2
  4. 2026OR-Transformer: Scaling Real-Time Decision-Making to 1,000 Itemscited by 1
  5. 2026Learning to Sell: Reinforcement Learning for Strategic Large Language Model Agents in Multi-Product Marketscited by 1
  6. 2026Pessimistic Minimax Learning for Public-Private Information Games under Unilateral Coveragecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluation Metrics for Safe Reinforcement Learning
  2. 2025Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewards
  3. 2025Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning
  4. 2024Revisiting Safe Exploration in Safe Reinforcement learning
  5. 2025Provably Optimal Reinforcement Learning under Safety Filtering
  6. 2024Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation
  7. 2024Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation
  8. 2026CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning