# Turning Safety into Competence: Minimally Exploitable Robot Policies via Safety-Filtered Reinforcement Learning

arXiv: https://arxiv.org/abs/2609.27312 · HTML: https://mlsanity.com/paper/2609.27312/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 25 of 25)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?](https://mlsanity.com/paper/2011.09533/) (2020)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Hamilton-Jacobi Reachability: A Brief Overview and Recent Advances](https://mlsanity.com/paper/1709.07523/) (2017)
- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)
- [Learning Control Barrier Functions from Expert Demonstrations](https://mlsanity.com/paper/2004.03315/) (2020)
- [DeepReach: A Deep Learning Approach to High-Dimensional Reachability](https://mlsanity.com/paper/2011.02082/) (2020)
- [The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systems](https://mlsanity.com/paper/2309.05837/) (2023)
- [Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysis](https://mlsanity.com/paper/2502.00935/) (2025)
- [Efficient Iterative Linear-Quadratic Approximations for Nonlinear Multi-Player General-Sum Differential Games](https://mlsanity.com/paper/1909.04694/) (2019)
- [Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failures](https://mlsanity.com/paper/2505.00779/) (2025)
- [ISAACS: Iterative Soft Adversarial Actor-Critic for Safety](https://mlsanity.com/paper/2212.03228/) (2022)
- [Gameplay Filters: Robust Zero-Shot Safety through Adversarial Imagination](https://mlsanity.com/paper/2405.00846/) (2024)
- [Safe Exploration by Solving Early Terminated MDP](https://mlsanity.com/paper/2107.04200/) (2021)
- [Predictive control barrier functions: Enhanced safety mechanisms for learning-based control](https://mlsanity.com/paper/2105.10241/) (2021)
- [Learning Vision-based Pursuit-Evasion Robot Policies](https://mlsanity.com/paper/2308.16185/) (2023)
- [Safety with Agency: Human-Centered Safety Filter with Application to AI-Assisted Motorsports](https://mlsanity.com/paper/2504.11717/) (2025)
- [MAGICS: Adversarial RL with Minimax Actors Guided by Implicit Critic Stackelberg for Convergent Neural Synthesis of Robot Safety](https://mlsanity.com/paper/2409.13867/) (2024)
- [Blending Data-Driven Priors in Dynamic Games](https://mlsanity.com/paper/2402.14174/) (2024)
- [Synthesis and Deployment of Maximal Robust Control Barrier Functions through Adversarial Reinforcement Learning](https://mlsanity.com/paper/2604.13192/) (2026)
- [Provably Optimal Reinforcement Learning under Safety Filtering](https://mlsanity.com/paper/2510.18082/) (2025)

## Abstract (arXiv)

Robots deployed for competitive tasks must outmaneuver their opponents without sacrificing safety. Existing approaches, including safe reinforcement learning (RL), train a single policy to achieve task success and avoid failures simultaneously. This coupling can complicate training and leave the learned policy exploitable by deliberate attacks. We propose Safety to Competence (S2C), a two-stage RL framework that separates safety synthesis from competitive task learning. We formulate competitive interactions as safety-critical Markov games and prove that perfect filtering preserves policy non-exploitability when all players commit to safe maneuvers. S2C learns a robust safety filter via adversarial RL, embeds it in the environment during task policy training, and retains the same filter at deployment. In simulated touchdown games, S2C outperforms eight safe RL baselines, achieving the highest win rate and Elo rating, and the lowest exploitability. Hardware stress tests against a human opponent confirm S2C's competence.