mlsanityData to 7 Oct 2026

ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning

arXiv 2410.09486 ↗First posted 12 Oct 2024cs.LG cs.ROcited by 22

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) is ubiquitous in the development of modern AI systems. However, state-of-the-art RL agents require extensive, and potentially unsafe, interactions with their environments to learn effectively. These limitations confine RL agents to simulated environments, hindering their ability to learn directly in real-world settings. In this work, we present ActSafe, a novel model-based RL algorithm for safe and efficient exploration. ActSafe learns a well-calibrated probabilistic model of the system and plans optimistically w.r.t. the epistemic uncertainty about the unknown dynamics, while enforcing pessimism w.r.t. the safety constraints. Under regularity assumptions on the constraints and dynamics, we show that ActSafe guarantees safety during learning while also obtaining a near-optimal policy in finite time. In addition, we propose a practical variant of ActSafe that builds on latest model-based RL advancements and enables safe exploration even in high-dimensional settings such as visual control. We empirically show that ActSafe obtains state-of-the-art performance in difficult exploration tasks on standard safe deep RL benchmarks while ensuring safety during learning.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 59 of its 74 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

59 of 74 references.

References · 59

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  4. 2023Mastering Diverse Domains through World Modelscited by 1,193
  5. 2018DeepMind Control Suitecited by 935
  6. 2017Constrained Policy Optimizationcited by 928
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  8. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  9. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  10. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  11. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  12. 2018Reward Constrained Policy Optimizationcited by 397
38 more
  1. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  2. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  3. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  4. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  5. 2019Challenges of Real-World Reinforcement Learningcited by 282
  6. 2018Safe Exploration in Continuous Action Spacescited by 237
  7. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  8. 2020Exploration-Exploitation in Constrained MDPscited by 146
  9. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
  10. 2016Bayesian Optimization with Safety Constraints: Safe and Automatic Parameter Tuning in Roboticscited by 134
  11. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 128
  12. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  13. 2024Expressive Whole-Body Control for Humanoid Robotscited by 117
  14. 2018Learning-based Model Predictive Control for Safe Explorationcited by 116
  15. 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 109
  16. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 109
  17. 2020Conservative Safety Critics for Explorationcited by 107
  18. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 101
  19. 2018A predictive safety filter for learning-based control of constrained nonlinear dynamical systemscited by 91
  20. 2020Sample-efficient Cross-Entropy Method for Real-time Planningcited by 90
  21. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 87
  22. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
  23. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learningcited by 74
  24. 2019Adaptive and Safe Bayesian Optimization in High Dimensions via One-Dimensional Subspacescited by 73
  25. 2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 72
  26. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  27. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 63
  28. 2022Constrained Policy Optimization via Bayesian World Modelscited by 55
  29. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
  30. 2020Constrained Upper Confidence Reinforcement Learningcited by 46
  31. 2023Finetuning Offline World Models in the Real Worldcited by 39
  32. 2022Near-Optimal Sample Complexity Bounds for Constrained MDPscited by 36
  33. 2023Optimistic Active Exploration of Dynamical Systemscited by 32
  34. 2021GoSafe: Globally Optimal Safe Robot Learningcited by 27
  35. 2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 26
  36. 2019Safe Contextual Bayesian Optimization for Sustainable Room Temperature PID Control Tuningcited by 24
  37. 2024Truly No-Regret Learning in Constrained MDPscited by 23
  38. 2022GoSafeOpt: Scalable Safe Exploration for Global Optimization of Dynamical Systemscited by 17

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 12.

22 papers.

Sort:

By area:

  1. 2025Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failurescited by 30
  2. 2024Embedding Safety into RL: A New Take on Trust Region Methodscited by 9
  3. 2025SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfercited by 8
  4. 2025SOMBRL: Scalable and Optimistic Model-Based RLcited by 7
  5. 2026Safe Exploration via Policy Priorscited by 6
  6. 2025Central Path Proximal Policy Optimizationcited by 3
  7. 2026Stable Agentic Control: Tool-Mediated LLM Architecture for Autonomous Cyber Defensecited by 3
  8. 2025Adaptive Shielding for Safe Reinforcement Learning under Hidden-Parameter Dynamics Shiftscited by 2
  9. 2025Learning to Undo: Rollback-Augmented Reinforcement Learning with Reversibility Signalscited by 2
  10. 2025A Novel Framework for Uncertainty-Driven Adaptive Explorationcited by 1
  11. 2025Safe and Near-Optimal Control with Online Dynamics Learningcited by 1
  12. 2026Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbedcited by 1
10 more
  1. 2026Model-Based Reinforcement Learning for Control under Time-Varying Dynamicscited by 1
  2. 2026Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertaintycited by 1
  3. 2026Uncertainty-Aware Predictive Safety Filters for Probabilistic Neural Network Dynamicscited by 1
  4. 2025A robust and adaptive MPC formulation for Gaussian process modelscited by 0
  5. 2026Decoupling Task and Behavior: A Two-Stage Reward Curriculum in Reinforcement Learning for Roboticscited by 0
  6. 2026Global Optimality for Constrained Exploration via Penalty Regularizationcited by 0
  7. 2026Sampling-Based Safe Reinforcement Learningcited by 0
  8. 2026Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learningcited by 0
  9. 2026All Models are Wrong, Knowing Where is Useful: On Model Uncertainty in Reinforcement Learningcited by 0
  10. 2026Evaluation Metrics for Safe Reinforcement Learningcited by 0