mlsanityData to 14 Oct 2026

Imagine to Ensure Safety in Hierarchical Reinforcement Learning

arXiv 2606.22509 ↗First posted 21 Jun 2026cs.AIcited by 0

On the map: World models → Latent dynamics models and model-based RL

also in 1 more
Abstract (arXiv)

This work investigates the safe exploration problem in reinforcement learning, where an agent must maximize cumulative performance while simultaneously satisfying safety constraints. This challenge becomes even more pronounced in long-horizon tasks, where existing safe methods face fundamental limitations due to compounding estimation errors and restricted exploration capabilities. To address this problem, we propose a method that combines a learnable world model with two complementary policies a high-level policy and a low-level policy to promote safety at both hierarchical levels. The high-level policy generates intermediate subgoals that bias exploration toward safe regions, while the low-level policy uses imagined rollouts in the learned world model to reduce unsafe behaviors when reaching these subgoals. The proposed method was evaluated on challenging long-horizon navigation and manipulation tasks with high-dimensional action spaces, where it significantly outperforms existing Safe RL baselines in both success rate and strong empirical constraint satisfaction, consistently meeting the prescribed safety budget across seeds, while prior approaches fail to effectively solve these complex long-horizon scenarios.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 29 of its 38 references that are in the map. First posted on arXiv 21 Jun 2026.

Builds on

29 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023SafeDreamer: Safe Reinforcement Learning with World Models14 Jul
  2. 2023Mastering Diverse Domains through World Models10 Jan
  3. 2020Learning to Walk in the Real World with Minimal Human Effort20 Feb
  4. 2019When to Trust Your Model: Model-Based Policy Optimization19 Jun

Other references · 25

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  3. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  4. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  5. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  6. 2018Safe Exploration in Continuous Action Spacescited by 247
  7. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 145
  8. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  9. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 115
  10. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  11. 2021Safe Reinforcement Learning Using Robust Control Barrier Functionscited by 59
  12. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
13 more
  1. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 58
  2. 2023Safe Reinforcement Learning via Probabilistic Logic Shieldscited by 42
  3. 2020Verifiably Safe Exploration for End-to-End Reinforcement Learningcited by 34
  4. 2022Towards Safe Reinforcement Learning with a Safety Editor Policycited by 30
  5. 2022Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithmcited by 29
  6. 2018Hierarchical Reinforcement Learning with Hindsightcited by 26
  7. 2022Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representationcited by 11
  8. 2022Model-free Neural Lyapunov Control for Safe Robot Navigationcited by 6
  9. 2024Safe Reinforcement Learning in Black-Box Environments via Adaptive Shieldingcited by 6
  10. 2024Safe Reinforcement Learning with Free-form Natural Language Constraints and Pre-Trained Language Modelscited by 4
  11. 2021Risk Conditioned Neural Motion Planningcited by 3
  12. 2022Guiding Safe Exploration with Weakest Preconditionscited by 3
  13. 2023Imagination-Augmented Hierarchical Reinforcement Learning for Safe and Interactive Autonomous Driving in Urban Environmentscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting Safe Exploration in Safe Reinforcement learning
  2. 2025Safe Planning and Policy Optimization via World Model Learning
  3. 2025Safety Representations for Safer Policy Learning
  4. 2024A Safe Exploration Strategy for Model-free Task Adaptation in Safety-constrained Grid Environments
  5. 2024A Dynamic Safety Shield for Safe and Efficient Reinforcement Learning of Navigation Tasks
  6. 2021Safe Continuous Control with Constrained Model-Based Policy Optimization
  7. 2024Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learning
  8. 2025Learning to explore when mistakes are not allowed