mlsanityData to 7 Oct 2026

Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulation

arXiv 2405.20860 ↗First posted 31 May 2024cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Safe reinforcement learning (RL) is crucial for deploying RL agents in real-world applications, as it aims to maximize long-term rewards while satisfying safety constraints. However, safe RL often suffers from sample inefficiency, requiring extensive interactions with the environment to learn a safe policy. We propose Efficient Safe Policy Optimization (ESPO), a novel approach that enhances the efficiency of safe RL through sample manipulation. ESPO employs an optimization framework with three modes: maximizing rewards, minimizing costs, and balancing the trade-off between the two. By dynamically adjusting the sampling process based on the observed conflict between reward and safety gradients, ESPO theoretically guarantees convergence, optimization stability, and improved sample complexity bounds. Experiments on the Safety-MuJoCo and Omnisafe benchmarks demonstrate that ESPO significantly outperforms existing primal-based and primal-dual-based baselines in terms of reward maximization and constraint satisfaction. Moreover, ESPO achieves substantial gains in sample efficiency, requiring 25--29% fewer samples than baselines, and reduces training time by 21--38%.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 43 of its 63 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

43 of 63 references.

References · 43

  1. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  3. 2017Constrained Policy Optimizationcited by 928
  4. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  5. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  6. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  7. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  8. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  9. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  10. 2020Projection-Based Constrained Policy Optimizationcited by 195
  11. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  12. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
31 more
  1. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
  2. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 131
  3. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 128
  4. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 76
  5. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 55
  6. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 55
  7. 2023State-wise Safe Reinforcement Learning: A Surveycited by 48
  8. 2021Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challengescited by 40
  9. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  10. 2023Iterative Reachability Estimation for Safe Reinforcement Learningcited by 33
  11. 2017Adaptive Sampling Strategies for Stochastic Optimizationcited by 29
  12. 2018Stability-certified reinforcement learning: A control-theoretic perspectivecited by 29
  13. 2015Global convergence rate analysis of unconstrained optimization methods based on probabilistic modelscited by 27
  14. 2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 26
  15. 2024Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulationcited by 21
  16. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
  17. 2021Computationally Efficient Safe Reinforcement Learning for Power Systemscited by 17
  18. 2021Recurrent Neural Network Controllers Synthesis with Stability Guarantees for Partially Observed Systemscited by 15
  19. 2019Global Convergence Rate Analysis of a Generic Line Search Algorithm with Noisecited by 13
  20. 2024Safe Reinforcement Learning on the Constraint Manifold: Theory and Applicationscited by 12
  21. 2023Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Riskcited by 11
  22. 2024AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methodscited by 8
  23. 2016ASTRO-DF: A Class of Adaptive Sampling Trust-Region Algorithms for Derivative-Free Stochastic Optimizationcited by 7
  24. 2020Balancing Rates and Variance via Adaptive Batch-Size for Stochastic Optimization Problemscited by 6
  25. 2022SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisitioncited by 6
  26. 2024Derivative-Free Optimization via Adaptive Sampling Strategiescited by 6
  27. 2020Adaptive sampling strategies for risk-averse stochastic optimization with constraintscited by 4
  28. 2020Adaptive Stochastic Optimizationcited by 3
  29. 2020Constrained and Composite Optimization via Adaptive Sampling Methodscited by 3
  30. 2023Safe and Sample-efficient Reinforcement Learning for Clustered Dynamic Environmentscited by 3
  31. 2023Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesiscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2026Traversal-as-Policy: Log-Distilled Gated Behavior Trees as Externalized, Verifiable Policies for Safe, Robust, and Efficient Agentscited by 5
  2. 2025Adaptive Shielding for Safe Reinforcement Learning under Hidden-Parameter Dynamics Shiftscited by 2
  3. 2026Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbedcited by 1
  4. 2025Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RLcited by 0
  5. 2025Trustworthy and Explainable Deep Reinforcement Learning for Safe and Energy-Efficient Process Control: A Use Case in Industrial Compressed Air Systemscited by 0
  6. 2026Safe Reinforcement Learning with Preference-based Constraint Inferencecited by 0
  7. 2026Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learningcited by 0
  8. 2026Boundary-Seeking Policy Gradient for Safe Reinforcement Learningcited by 0