# Safe Reinforcement Learning Using Robust Control Barrier Functions

arXiv: https://arxiv.org/abs/2110.05415 · HTML: https://mlsanity.com/paper/2110.05415/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 11 Oct 2021.

## Builds on (most cited first, 26 of 26)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Control Barrier Function Based Quadratic Programs for Safety Critical Systems](https://mlsanity.com/paper/1609.06408/) (2016)
- [Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones](https://mlsanity.com/paper/2010.15920/) (2020)
- [A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems](https://mlsanity.com/paper/1705.01292/) (2017)
- [Lyapunov-based Safe Policy Optimization for Continuous Control](https://mlsanity.com/paper/1901.10031/) (2019)
- [Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions](https://mlsanity.com/paper/2004.07584/) (2020)
- [Safety-Critical Model Predictive Control with Discrete-Time Control Barrier Function](https://mlsanity.com/paper/2007.11718/) (2020)
- [Robustness of Control Barrier Functions for Safety Critical Control](https://mlsanity.com/paper/1612.01554/) (2016)
- [SAMBA: Safe Model-Based & Active Reinforcement Learning](https://mlsanity.com/paper/2006.09436/) (2020)
- [Control Barriers in Bayesian Learning of System Dynamics](https://mlsanity.com/paper/2012.14964/) (2020)
- [Safety-Critical Control using Optimal-decay Control Barrier Function with Guaranteed Point-wise Feasibility](https://mlsanity.com/paper/2103.12375/) (2021)
- [Robust Barrier Functions for a Fully Autonomous, Remotely Accessible Swarm-Robotics Testbed](https://mlsanity.com/paper/1909.02966/) (2019)
- [Data-Driven Robust Barrier Functions for Safe, Long-Term Operation](https://mlsanity.com/paper/2104.07592/) (2021)
- [Constraint Learning for Control Tasks with Limited Duration Barrier Functions](https://mlsanity.com/paper/1908.09506/) (2019)
- [Persistification of Robotic Tasks](https://mlsanity.com/paper/1903.05810/) (2019)

## Built on by (most cited first, 30 of 59)

- [A Survey on Physics Informed Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2309.01909/) (2023)
- [CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions](https://mlsanity.com/paper/2510.14959/) (2025)
- [Safe Reinforcement Learning on the Constraint Manifold: Theory and Applications](https://mlsanity.com/paper/2404.09080/) (2024)
- [Stable and Safe Reinforcement Learning via a Barrier-Lyapunov Actor-Critic Approach](https://mlsanity.com/paper/2304.04066/) (2023)
- [Safety Filters for Black-Box Dynamical Systems by Learning Discriminating Hyperplanes](https://mlsanity.com/paper/2402.05279/) (2024)
- [Discrete GCBF Proximal Policy Optimization for Multi-agent Safe Optimal Control](https://mlsanity.com/paper/2502.03640/) (2025)
- [Multi-Agent Reinforcement Learning Guided by Signal Temporal Logic Specifications](https://mlsanity.com/paper/2306.06808/) (2023)
- [Towards Safe Autonomous Driving Policies using a Neuro-Symbolic Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2307.01316/) (2023)
- [Verified Safe Reinforcement Learning for Neural Network Dynamic Models](https://mlsanity.com/paper/2405.15994/) (2024)
- [Hierarchical Multi-Agent Reinforcement Learning with Control Barrier Functions for Safety-Critical Autonomous Systems](https://mlsanity.com/paper/2507.14850/) (2025)
- [On the Optimality, Stability, and Feasibility of Control Barrier Functions: An Adaptive Learning-Based Approach](https://mlsanity.com/paper/2305.03608/) (2023)
- [Bridging the gap between Learning-to-plan, Motion Primitives and Safe Reinforcement Learning](https://mlsanity.com/paper/2408.14063/) (2024)
- [A Survey of Reinforcement Learning for Optimization in Automation](https://mlsanity.com/paper/2502.09417/) (2025)
- [Improved Robustness of Deep Reinforcement Learning for Control of Time-Varying Systems by Bounded Extremum Seeking](https://mlsanity.com/paper/2510.02490/) (2025)
- [Learning Adaptive Safety for Multi-Agent Systems](https://mlsanity.com/paper/2309.10657/) (2023)
- [Learning Predictive Safety Filter via Decomposition of Robust Invariant Set](https://mlsanity.com/paper/2311.06769/) (2023)
- [Implicit Safe Set Algorithm for Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2405.02754/) (2024)
- [Provably Optimal Reinforcement Learning under Safety Filtering](https://mlsanity.com/paper/2510.18082/) (2025)
- [Machine Learning Meets Advanced Robotic Manipulation](https://mlsanity.com/paper/2309.12560/) (2023)
- [WROOM: An Autonomous Driving Approach for Off-Road Navigation](https://mlsanity.com/paper/2404.08855/) (2024)
- [Toward Scalable Multirobot Control: Fast Policy Learning in Distributed MPC](https://mlsanity.com/paper/2412.19669/) (2024)
- [Back to Base: Towards Hands-Off Learning via Safe Resets with Reach-Avoid Safety Filters](https://mlsanity.com/paper/2501.02620/) (2025)
- [Certificated Actor-Critic: Hierarchical Reinforcement Learning with Control Barrier Functions for Safe Navigation](https://mlsanity.com/paper/2501.17424/) (2025)
- [T-CBF: Traversability-based Control Barrier Function to Navigate Vertically Challenging Terrain](https://mlsanity.com/paper/2503.06083/) (2025)
- [Multi-Constraint Safe Reinforcement Learning via Closed-form Solution for Log-Sum-Exp Approximation of Control Barrier Functions](https://mlsanity.com/paper/2505.00671/) (2025)
- [End-to-End Humanoid Robot Safe and Comfortable Locomotion Policy](https://mlsanity.com/paper/2508.07611/) (2025)
- [Safety Filter Design for Neural Network Systems via Convex Optimization](https://mlsanity.com/paper/2308.08086/) (2023)
- [Towards Optimal Head-to-head Autonomous Racing with Curriculum Reinforcement Learning](https://mlsanity.com/paper/2308.13491/) (2023)
- [Distributionally Safe Reinforcement Learning under Model Uncertainty: A Single-Level Approach by Differentiable Convex Programming](https://mlsanity.com/paper/2310.02459/) (2023)
- [The Feasibility Theory of Constrained Reinforcement Learning: A Tutorial Study](https://mlsanity.com/paper/2404.10064/) (2024)

## Abstract (arXiv)

Reinforcement Learning (RL) has been shown to be effective in many scenarios. However, it typically requires the exploration of a sufficiently large number of state-action pairs, some of which may be unsafe. Consequently, its application to safety-critical systems remains a challenge. An increasingly common approach to address safety involves the addition of a safety layer that projects the RL actions onto a safe set of actions. In turn, a difficulty for such frameworks is how to effectively couple RL with the safety layer to improve the learning performance. In this paper, we frame safety as a differentiable robust-control-barrier-function layer in a model-based RL framework. Moreover, we also propose an approach to modularly learn the underlying reward-driven task, independent of safety constraints. We demonstrate that this approach both ensures safety and effectively guides exploration during training in a range of experiments, including zero-shot transfer when the reward is learned in a modular way.