mlsanityData to 7 Oct 2026

Safe Deep Reinforcement Learning by Verifying Task-Level Properties

arXiv 2302.10030 ↗First posted 20 Feb 2023cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Cost functions are commonly employed in Safe Deep Reinforcement Learning (DRL). However, the cost is typically encoded as an indicator function due to the difficulty of quantifying the risk of policy decisions in the state space. Such an encoding requires the agent to visit numerous unsafe states to learn a cost-value function to drive the learning process toward safety. Hence, increasing the number of unsafe interactions and decreasing sample efficiency. In this paper, we investigate an alternative approach that uses domain knowledge to quantify the risk in the proximity of such states by defining a violation metric. This metric is computed by verifying task-level properties, shaped as input-output conditions, and it is used as a penalty to bias the policy away from unsafe states without learning an additional value function. We investigate the benefits of using the violation metric in standard Safe DRL benchmarks and robotic mapless navigation tasks. The navigation experiments bridge the gap between Safe DRL and robotics, introducing a framework that allows rapid testing on real robots. Our experiments show that policies trained with the violation penalty achieve higher performance over Safe DRL baselines and significantly reduce the number of visited unsafe states.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 29 of its 52 references that are in the map. First posted on arXiv 20 Feb 2023.

Builds on

29 of 52 references.

References · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Constrained Policy Optimizationcited by 928
  3. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  4. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 739
  5. 2018Reward Constrained Policy Optimizationcited by 397
  6. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  7. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  8. 2019Algorithms for Verifying Deep Neural Networkscited by 221
  9. 2020Projection-Based Constrained Policy Optimizationcited by 195
  10. 2018Formal Security Analysis of Neural Networks using Symbolic Intervalscited by 185
  11. 2020Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifierscited by 183
  12. 2018Efficient Formal Safety Analysis of Neural Networkscited by 165
17 more
  1. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
  2. 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
  3. 2017Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigationcited by 90
  4. 2021On the Expressivity of Markov Rewardcited by 71
  5. 2018Learning Navigation Behaviors End-to-End with AutoRLcited by 70
  6. 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65
  7. 2021Safe Multi-Agent Reinforcement Learning via Shieldingcited by 60
  8. 2016Deep Reinforcement Learning with Successor Features for Navigation across Similar Environmentscited by 50
  9. 2022Verifying Learning-Based Robotic Navigation Systemscited by 33
  10. 2021Direct Behavior Specification via Constrained Reinforcement Learningcited by 30
  11. 2021Safe Reinforcement Learning using Formal Verification for Tissue Retraction in Autonomous Robotic-Assisted Surgerycited by 22
  12. 2023The #DNN-Verification Problem: Counting Unsafe Inputs for Deep Neural Networkscited by 19
  13. 2020Model-based Reinforcement Learning from Signal Temporal Logic Specificationscited by 18
  14. 2021Centralizing State-Values in Dueling Networks for Multi-Robot Reinforcement Learning Mapless Navigationcited by 13
  15. 2021Curriculum Learning for Safe Mapless Navigationcited by 11
  16. 2021Benchmarking Safe Deep Reinforcement Learning in Aquatic Navigationcited by 10
  17. 2023Online Safety Property Collection and Refinement for Safe Deep Reinforcement Learning in Mapless Navigationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 2; 2026: 2.

9 papers.

  1. 2023Enumerating Safe Regions in Deep Neural Networks with Provable Probabilistic Guaranteescited by 19
  2. 2023SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agentscited by 9
  3. 2025RL2Grid: Benchmarking Reinforcement Learning in Power Grid Operationscited by 8
  4. 2024Improving Policy Optimization via varepsilon-Retraincited by 5
  5. 2024On Stateful Value Factorization in Multi-Agent Reinforcement Learningcited by 5
  6. 2025Designing Control Barrier Function via Probabilistic Enumeration for Safe Reinforcement Learning Navigationcited by 3
  7. 2023Scaling #DNN-Verification Tools with Efficient Bound Propagation and Parallel Computingcited by 1
  8. 2026Probabilistic Verification of Recurrent Neural Networks for Single and Multi-Agent Reinforcement Learningcited by 0
  9. 2026A Survey on the Verification of Reinforcement Learning Policiescited by 0