mlsanityData to 14 Oct 2026

Safe Policy Exploration Improvement via Subgoals

arXiv 2408.13881 ↗First posted 25 Aug 2024cs.RO cs.LGcited by 0

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Reinforcement learning is a widely used approach to autonomous navigation, showing potential in various tasks and robotic setups. Still, it often struggles to reach distant goals when safety constraints are imposed (e.g., the wheeled robot is prohibited from moving close to the obstacles). One of the main reasons for poor performance in such setups, which is common in practice, is that the need to respect the safety constraints degrades the exploration capabilities of an RL agent. To this end, we introduce a novel learnable algorithm that is based on decomposing the initial problem into smaller sub-problems via intermediate goals, on the one hand, and respects the limit of the cumulative safety constraints, on the other hand -- SPEIS(Safe Policy Exploration Improvement via Subgoals). It comprises the two coupled policies trained end-to-end: subgoal and safe. The subgoal policy is trained to generate the subgoal based on the transitions from the buffer of the safe (main) policy that helps the safe policy to reach distant goals. Simultaneously, the safe policy maximizes its rewards while attempting not to violate the limit of the cumulative safety constraints, thus providing a certain level of safety. We evaluate SPEIS in a wide range of challenging (simulated) environments that involve different types of robots in two different environments: autonomous vehicles from the POLAMP environment and car, point, doggo, and sweep from the safety-gym environment. We demonstrate that our method consistently outperforms state-of-the-art competitors and can significantly reduce the collision rate while maintaining high success rates (higher by 80% compared to the best-performing methods).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 17 of its 32 references that are in the map. First posted on arXiv 25 Aug 2024.

Builds on

17 of 32 references.

References · 17

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2017Hindsight Experience Replaycited by 1,136
  3. 2017Constrained Policy Optimizationcited by 983
  4. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  5. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  6. 2016A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehiclescited by 254
  7. 2018Safe Exploration in Continuous Action Spacescited by 247
  8. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  9. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  10. 2020Learning to Walk in the Real World with Minimal Human Effortcited by 111
  11. 2020Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehiclescited by 110
  12. 2021A Survey of Deep RL and IL for Autonomous Driving Policy Learningcited by 61
5 more
  1. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
  2. 2022Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithmcited by 29
  3. 2017Learning Multi-Level Hierarchies with Hindsightcited by 20
  4. 2021Risk Conditioned Neural Motion Planningcited by 3
  5. 2022Guiding Safe Exploration with Weakest Preconditionscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Dynamic Safety Shield for Safe and Efficient Reinforcement Learning of Navigation Tasks
  2. 2026Imagine to Ensure Safety in Hierarchical Reinforcement Learning
  3. 2025Safe Multi-Agent Navigation guided by Goal-Conditioned Safe Reinforcement Learning
  4. 2026Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning
  5. 2026Safety-Constrained Reinforcement Learning with Post-Training Reachability Verification for Robot Navigation
  6. 2024Safety Optimized Reinforcement Learning via Multi-Objective Policy Optimization
  7. 2026CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning
  8. 2024Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learning