# Stable and Safe Reinforcement Learning via a Barrier-Lyapunov Actor-Critic Approach

arXiv: https://arxiv.org/abs/2304.04066 · HTML: https://mlsanity.com/paper/2304.04066/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 8 Apr 2023.

## Builds on (most cited first, 26 of 26)

- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Projection-Based Constrained Policy Optimization](https://mlsanity.com/paper/2010.03152/) (2020)
- [Constrained Reinforcement Learning Has Zero Duality Gap](https://mlsanity.com/paper/1910.13393/) (2019)
- [Safe Control with Learned Certificates: A Survey of Neural Lyapunov, Barrier, and Contraction methods](https://mlsanity.com/paper/2202.11762/) (2022)
- [Learning-based Model Predictive Control for Safe Exploration](https://mlsanity.com/paper/1803.08287/) (2018)
- [Safe Nonlinear Control Using Robust Neural Lyapunov-Barrier Functions](https://mlsanity.com/paper/2109.06697/) (2021)
- [Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions](https://mlsanity.com/paper/2004.07584/) (2020)
- [Safe Reinforcement Learning Using Robust Control Barrier Functions](https://mlsanity.com/paper/2110.05415/) (2021)
- [Actor-Critic Reinforcement Learning for Control with Stability Guarantee](https://mlsanity.com/paper/2004.14288/) (2020)
- [Stabilizing Neural Control Using Self-Learned Almost Lyapunov Critics](https://mlsanity.com/paper/2107.04989/) (2021)
- [Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safety](https://mlsanity.com/paper/2105.10682/) (2021)
- [Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learning](https://mlsanity.com/paper/2111.07695/) (2021)
- [Learning Safe, Generalizable Perception-based Hybrid Control with Certificates](https://mlsanity.com/paper/2201.00932/) (2022)
- [Safe Control Algorithms Using Energy Functions: A Unified Framework, Benchmark, and New Directions](https://mlsanity.com/paper/1908.01883/) (2019)
- [QuaDUE-CCM: Interpretable Distributional Reinforcement Learning using Uncertain Contraction Metrics for Precise Quadrotor Trajectory Tracking](https://mlsanity.com/paper/2207.07789/) (2022)
- [Constrained Reinforcement Learning using Distributional Representation for Trustworthy Quadrotor UAV Tracking Control](https://mlsanity.com/paper/2302.11694/) (2023)
- [Learn Zero-Constraint-Violation Policy in Model-Free Constrained Reinforcement Learning](https://mlsanity.com/paper/2111.12953/) (2021)
- [Safety Verification and Controller Synthesis for Systems with Input Constraints](https://mlsanity.com/paper/2204.09386/) (2022)
- [On the Undesired Equilibria Induced by Control Barrier Function Based Quadratic Programs](https://mlsanity.com/paper/2104.14895/) (2021)
- [Physical Deep Reinforcement Learning Towards Safety Guarantee](https://mlsanity.com/paper/2303.16860/) (2023)

## Built on by (most cited first, 13 of 13)

- [SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphs](https://mlsanity.com/paper/2404.00469/) (2024)
- [Physics-Regulated Deep Reinforcement Learning: Invariant Embeddings](https://mlsanity.com/paper/2305.16614/) (2023)
- [A Review On Safe Reinforcement Learning Using Lyapunov and Barrier Functions](https://mlsanity.com/paper/2508.09128/) (2025)
- [Debiasing Machine Unlearning with Counterfactual Examples](https://mlsanity.com/paper/2404.15760/) (2024)
- [Exploring CausalWorld: Enhancing robotic manipulation via knowledge transfer and curriculum learning](https://mlsanity.com/paper/2403.17266/) (2024)
- [Simplex-enabled Safe Continual Learning Machine](https://mlsanity.com/paper/2409.05898/) (2024)
- [Off Policy Lyapunov Stability in Reinforcement Learning](https://mlsanity.com/paper/2509.09863/) (2025)
- [Physics-model-guided Worst-case Sampling for Safe Reinforcement Learning](https://mlsanity.com/paper/2412.13224/) (2024)
- [SALSA-RL: Stability Analysis in the Latent Space of Actions for Reinforcement Learning](https://mlsanity.com/paper/2502.15512/) (2025)
- [A Novel Multi-Timescale Stability-Preserving Hierarchical Reinforcement Learning Controller Framework for Adaptive Control in High-Dimensional Dynamical Systems](https://mlsanity.com/paper/2510.22420/) (2025)
- [Real-DRL: Teach and Learn in Reality](https://mlsanity.com/paper/2511.00112/) (2025)
- [IL-ACT: Imitation Learning with Adaptive Cartesian Tracking Control for a 30-ton Excavator](https://mlsanity.com/paper/2609.16696/) (2026)
- [CALOS: Control-Affine Lyapunov On-manifold Safety Layer for Safe Deep Reinforcement Learning for Quadrotors](https://mlsanity.com/paper/2609.17758/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) has demonstrated impressive performance in various areas such as video games and robotics. However, ensuring safety and stability, which are two critical properties from a control perspective, remains a significant challenge when using RL to control real-world systems. In this paper, we first provide definitions of safety and stability for the RL system, and then combine the control barrier function (CBF) and control Lyapunov function (CLF) methods with the actor-critic method in RL to propose a Barrier-Lyapunov Actor-Critic (BLAC) framework which helps maintain the aforementioned safety and stability for the system. In this framework, CBF constraints for safety and CLF constraint for stability are constructed based on the data sampled from the replay buffer, and the augmented Lagrangian method is used to update the parameters of the RL-based controller. Furthermore, an additional backup controller is introduced in case the RL-based controller cannot provide valid control signals when safety and stability constraints cannot be satisfied simultaneously. Simulation results show that this framework yields a controller that can help the system approach the desired state and cause fewer violations of safety constraints compared to baseline algorithms.