mlsanityData to 7 Oct 2026

Safe Exploration in Continuous Action Spaces

arXiv 1801.08757 ↗First posted 26 Jan 2018cs.AIcited by 237

Not on the map. Why?

Abstract (arXiv)

We address the problem of deploying a reinforcement learning (RL) agent on a physical system such as a datacenter cooling unit or robot, where critical constraints must never be violated. We show how to exploit the typically smooth dynamics of these systems and enable RL algorithms to never violate constraints during learning. Our technique is to directly add to the policy a safety layer that analytically solves an action correction formulation per each state. The novelty of obtaining an elegant closed-form solution is attained due to a linearized model, learned on past trajectories consisting of arbitrary actions. This is to mimic the real-world circumstances where data logs were generated with a behavior policy that is implausible to describe mathematically; such cases render the known safety-aware off-policy methods inapplicable. We demonstrate the efficacy of our approach on new representative physics-based environments, and prevail where reward shaping fails by maintaining zero constraint violations.

Key facts. Cited by 237 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Jan 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 41; 2022: 49; 2023: 39; 2024: 43; 2025: 27; 2026: 37.

237 papers.

Sort:

By area:

  1. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  2. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  3. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  4. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
  5. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  6. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 69
  7. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
  8. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  9. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  10. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 63
  11. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
  12. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
38 more
  1. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  2. 2022Constrained Policy Optimization via Bayesian World Modelscited by 55
  3. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 55
  4. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 54
  5. 2024Agile But Safe: Learning Collision-Free High-Speed Legged Locomotioncited by 53
  6. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  7. 2021CombOptNet: Fit the Right NP-Hard Problem by Learning Integer Programming Constraintscited by 49
  8. 2023State-wise Safe Reinforcement Learning: A Surveycited by 48
  9. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 45
  10. 2021safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Roboticscited by 42
  11. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 42
  12. 2023Safe Reinforcement Learning via Probabilistic Logic Shieldscited by 41
  13. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  14. 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
  15. 2021Context-Aware Safe Reinforcement Learning for Non-Stationary Environmentscited by 33
  16. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  17. 2022Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarkingcited by 31
  18. 2021Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safetycited by 30
  19. 2021Direct Behavior Specification via Constrained Reinforcement Learningcited by 30
  20. 2021Safe Driving via Expert Guided Policy Optimizationcited by 29
  21. 2022Safe Reinforcement Learning for Legged Locomotioncited by 29
  22. 2022Towards Safe Reinforcement Learning with a Safety Editor Policycited by 28
  23. 2022Sustainable Online Reinforcement Learning for Auto-biddingcited by 28
  24. 2022Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithmcited by 27
  25. 2022Safe Control Under Input Limits with Neural Control Barrier Functionscited by 27
  26. 2021Density Constrained Reinforcement Learningcited by 25
  27. 2021Differentiable Trust Region Layers for Deep Reinforcement Learningcited by 24
  28. 2021Stability Verification in Stochastic Control Systems via Neural Network Supermartingalescited by 24
  29. 2022Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guaranteescited by 24
  30. 2022Safe Reinforcement Learning Using Black-Box Reachability Analysiscited by 22
  31. 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 22
  32. 2022Learning Control Policies for Stochastic Systems with Reach-avoid Guaranteescited by 21
  33. 2022Asynchronous Reinforcement Learning for Real-Time Control of Physical Robotscited by 19
  34. 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 19
  35. 2021Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimizationcited by 18
  36. 2021Faster Algorithm and Sharper Analysis for Constrained Markov Decision Processcited by 18
  37. 2021Safe Reinforcement Learning with Chance-constrained Model Predictive Controlcited by 18
  38. 2022Evaluating Model-free Reinforcement Learning toward Safety-critical Taskscited by 18

50 most cited of 237 All 237 (JSON)