mlsanityData to 7 Oct 2026

A Safe Exploration Strategy for Model-free Task Adaptation in Safety-constrained Grid Environments

arXiv 2408.00997 ↗First posted 2 Aug 2024cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Training a model-free reinforcement learning agent requires allowing the agent to sufficiently explore the environment to search for an optimal policy. In safety-constrained environments, utilizing unsupervised exploration or a non-optimal policy may lead the agent to undesirable states, resulting in outcomes that are potentially costly or hazardous for both the agent and the environment. In this paper, we introduce a new exploration framework for navigating the grid environments that enables model-free agents to interact with the environment while adhering to safety constraints. Our framework includes a pre-training phase, during which the agent learns to identify potentially unsafe states based on both observable features and specified safety constraints in the environment. Subsequently, a binary classification model is trained to predict those unsafe states in new environments that exhibit similar dynamics. This trained classifier empowers model-free agents to determine situations in which employing random exploration or a suboptimal policy may pose safety risks, in which case our framework prompts the agent to follow a predefined safe policy to mitigate the potential for hazardous consequences. We evaluated our framework on three randomly generated grid environments and demonstrated how model-free agents can safely adapt to new tasks and learn optimal policies for new environments. Our results indicate that by defining an appropriate safe policy and utilizing a well-trained model to detect unsafe states, our framework enables a model-free agent to adapt to new tasks and environments with significantly fewer safety violations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 46 references that are in the map. First posted on arXiv 2 Aug 2024.

Builds on

27 of 46 references.

References · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  3. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  4. 2017Safe Reinforcement Learning via Shieldingcited by 427
  5. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  6. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  7. 2017Hamilton-Jacobi Reachability: A Brief Overview and Recent Advancescited by 186
  8. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  9. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 150
  10. 2017Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Gamescited by 93
  11. 2019Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systemscited by 90
  12. 2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 72
15 more
  1. 2023State-wise Safe Reinforcement Learning: A Surveycited by 48
  2. 2021Safety and Liveness Guarantees through Reach-Avoid Reinforcement Learningcited by 43
  3. 2020Cautious Adaptation For Reinforcement Learning in Safety-Critical Settingscited by 42
  4. 2014Safe Exploration of State and Action Spaces in Reinforcement Learningcited by 37
  5. 2018Deep Reinforcement Learning for General Video Game AIcited by 25
  6. 2019Deep Reinforcement Learning with Enhanced Safety for Autonomous Highway Drivingcited by 21
  7. 2019An Efficient Reachability-Based Framework for Provably Safe Autonomous Navigation in Unknown Environmentscited by 18
  8. 2022Evaluating Model-free Reinforcement Learning toward Safety-critical Taskscited by 18
  9. 2022Safe Reinforcement Learning of Dynamic High-Dimensional Robotic Tasks: Navigation, Manipulation, Interactioncited by 15
  10. 2020Deep Reinforcement learning for real autonomous mobile robot navigation in indoor environmentscited by 13
  11. 2018Modular Architecture for StarCraft II with Deep Reinforcement Learningcited by 10
  12. 2021Reinforcement Learning to Optimize Lifetime Value in Cold-Start Recommendationcited by 9
  13. 2017Correlational Dueling Bandits with Application to Clinical Treatment in Large Decision Spacescited by 6
  14. 2022Infinite-Horizon Reach-Avoid Zero-Sum Games via Deep Reinforcement Learningcited by 6
  15. 2018A Survey of Deep Learning Techniques for Mobile Robot Applicationscited by 3

Built on by

Not cited yet.