mlsanityData to 14 Oct 2026

SPoRt -- Safe Policy Ratio: Certified Training and Deployment of Task Policies in Model-Free RL

arXiv 2504.06386 ↗First posted 8 Apr 2025cs.LGcited by 1

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

To apply reinforcement learning to safety-critical applications, we ought to provide safety guarantees during both policy training and deployment. In this work, we present theoretical results that place a bound on the probability of violating a safety property for a new task-specific policy in a model-free, episodic setting. This bound, based on a maximum policy ratio computed with respect to a 'safe' base policy, can also be applied to temporally-extended properties (beyond safety) and to robust control problems. To utilize these results, we introduce SPoRt, which provides a data-driven method for computing this bound for the base policy using the scenario approach, and includes Projected PPO, a new projection-based approach for training the task-specific policy while maintaining a user-specified bound on property violation. SPoRt thus enables users to trade off safety guarantees against task-specific performance. Complementing our theoretical results, we present experimental results demonstrating this trade-off and comparing the theoretical bound to posterior bounds derived from empirical violation rates.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 23 of its 44 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

23 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Embedding Safety into RL: A New Take on Trust Region Methods5 Nov

Other references · 22

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2015Prioritized Experience Replaycited by 1,099
  5. 2017Constrained Policy Optimizationcited by 983
  6. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
  7. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  8. 2017Safe Reinforcement Learning via Shieldingcited by 449
  9. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  10. 2017A Rewriting System for Convex Optimization Problemscited by 179
  11. 2019Multi-Agent Deep Reinforcement Learning for Large-scale Traffic Signal Controlcited by 164
  12. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
10 more
  1. 2023The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systemscited by 88
  2. 2019Safe Reinforcement Learning on Autonomous Vehiclescited by 50
  3. 2020Reinforcement Learning for Autonomous Driving with Latent State Inference and Spatial-Temporal Relationshipscited by 37
  4. 2019Certified Reinforcement Learning with Logic Guidancecited by 36
  5. 2022Online Shielding for Reinforcement Learningcited by 22
  6. 2019A Review of Reinforcement Learning for Autonomous Building Energy Managementcited by 21
  7. 2018Metrics for Signal Temporal Logic Formulaecited by 9
  8. 2015Quantitative Approximation of the Probability Distribution of a Markov Process by Formal Abstractionscited by 1
  9. 2023State Aggregation for Distributed Value Iteration in Dynamic Programmingcited by 1
  10. 2024Finite sample learning of moving targetscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SafeAdapt: Provably Safe Policy Updates in Deep Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewards
  2. 2024Safety through Permissibility: Shield Construction for Fast and Safe Reinforcement Learning
  3. 2026Turning Safety into Competence: Minimally Exploitable Robot Policies via Safety-Filtered Reinforcement Learning
  4. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints
  5. 2026Evaluation Metrics for Safe Reinforcement Learning
  6. 2025CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions
  7. 2024A Survey of Constraint Formulations in Safe Reinforcement Learning
  8. 2024Progressive Safeguards for Safe and Model-Agnostic Reinforcement Learning