# Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents

arXiv: https://arxiv.org/abs/2410.11671 · HTML: https://mlsanity.com/paper/2410.11671/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Safe Exploration in Continuous Action Spaces](https://mlsanity.com/paper/1801.08757/) (2018)
- [A predictive safety filter for learning-based control of constrained nonlinear dynamical systems](https://mlsanity.com/paper/1812.05506/) (2018)
- [The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systems](https://mlsanity.com/paper/2309.05837/) (2023)
- [OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World](https://mlsanity.com/paper/1709.07643/) (2017)
- [acados: a modular open-source framework for fast embedded optimal control](https://mlsanity.com/paper/1910.13753/) (2019)
- [safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Robotics](https://mlsanity.com/paper/2109.06325/) (2021)
- [Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking](https://mlsanity.com/paper/2205.06750/) (2022)
- [Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopes](https://mlsanity.com/paper/2210.10691/) (2022)
- [Safe Reinforcement Learning with Chance-constrained Model Predictive Control](https://mlsanity.com/paper/2112.13941/) (2021)
- [A computationally efficient robust model predictive control framework for uncertain nonlinear systems -- extended version](https://mlsanity.com/paper/1910.12081/) (2019)
- [Zeus: A System Description of the Two-Time Winner of the Collegiate SAE AutoDrive Competition](https://mlsanity.com/paper/2004.08752/) (2020)
- [Differentiable Safe Controller Design through Control Barrier Functions](https://mlsanity.com/paper/2209.10034/) (2022)
- [Design of a Trajectory Tracking Controller for a Nanoquadcopter](https://mlsanity.com/paper/1608.05786/) (2016)
- [Multi-Step Model Predictive Safety Filters: Reducing Chattering by Increasing the Prediction Horizon](https://mlsanity.com/paper/2309.11453/) (2023)
- [Learn With Imagination: Safe Set Guided State-wise Constrained Policy Optimization](https://mlsanity.com/paper/2308.13140/) (2023)
- [A Remote Sim2real Aerial Competition: Fostering Reproducibility and Solutions' Diversity in Robotics Challenges](https://mlsanity.com/paper/2308.16743/) (2023)

## Built on by (most cited first, 17 of 17)

- [CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions](https://mlsanity.com/paper/2510.14959/) (2025)
- [A view on learning robust goal-conditioned value functions: Interplay between RL and MPC](https://mlsanity.com/paper/2502.06996/) (2025)
- [Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?](https://mlsanity.com/paper/2509.12833/) (2025)
- [Provably Optimal Reinforcement Learning under Safety Filtering](https://mlsanity.com/paper/2510.18082/) (2025)
- [Hard Constraints Meet Soft Generation: Guaranteed Feasibility for LLM-based Combinatorial Optimization](https://mlsanity.com/paper/2602.01090/) (2026)
- [X-Safe: Embodiment-Agnostic Safeguarding for Robot Arm Manipulation Policies via Action Masking](https://mlsanity.com/paper/2606.22278/) (2026)
- [Leveraging Analytic Gradients in Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2506.01665/) (2025)
- [Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty](https://mlsanity.com/paper/2604.25508/) (2026)
- [Uncertainty-Aware Predictive Safety Filters for Probabilistic Neural Network Dynamics](https://mlsanity.com/paper/2604.26836/) (2026)
- [Permissive Safety Through Trusted Inference: Verifiable Belief-Space Neural Safety Filters for Assured Interactive Robotics](https://mlsanity.com/paper/2606.02562/) (2026)
- [Safe Reinforcement Learning with a Predictive Safety Filter for Motion Planning and Control: A Drifting Vehicle Example](https://mlsanity.com/paper/2506.22894/) (2025)
- [LEARN: Learning End-to-End Aerial Resource-Constrained Multi-Robot Navigation](https://mlsanity.com/paper/2511.17765/) (2025)
- [Reversible Simplex Supervision with Post-Action Debt Accounting for Goal-Reaching RL](https://mlsanity.com/paper/2601.19499/) (2026)
- [All Models are Wrong, Knowing Where is Useful: On Model Uncertainty in Reinforcement Learning](https://mlsanity.com/paper/2606.01363/) (2026)
- [DODGER: Safety-Guided Reinforcement Learning for Robot Navigation Among Dynamic Obstacles](https://mlsanity.com/paper/2609.38873/) (2026)
- [Beyond Safe Execution: Safe Online Reinforcement Learning for Autonomous Driving with Value Calibration and Adaptive Guidance](https://mlsanity.com/paper/2610.17799/) (2026)
- [Safe Reinforcement Learning with Receding-Horizon Safety Critic and Differentiable Action Shields](https://mlsanity.com/paper/2610.20255/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) controllers are flexible and performant but rarely guarantee safety. Safety filters impart hard safety guarantees to RL controllers while maintaining flexibility. However, safety filters can cause undesired behaviours due to the separation between the controller and the safety filter, often degrading performance and robustness. In this paper, we analyze several modifications to incorporating the safety filter in training RL controllers rather than solely applying it during evaluation. The modifications allow the RL controller to learn to account for the safety filter, improving performance. This paper presents a comprehensive analysis of training RL with safety filters, featuring simulated and real-world experiments with a Crazyflie 2.0 drone. We examine how various training modifications and hyperparameters impact performance, sample efficiency, safety, and chattering. Our findings serve as a guide for practitioners and researchers focused on safety filters and safe RL.