# Actor-Critic Reinforcement Learning for Control with Stability Guarantee

arXiv: https://arxiv.org/abs/2004.14288 · HTML: https://mlsanity.com/paper/2004.14288/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 30 of 45)

- [SneakyPrompt: Jailbreaking Text-to-image Generative Models](https://mlsanity.com/paper/2305.12082/) (2023)
- [Neural Lyapunov Control for Discrete-Time Systems](https://mlsanity.com/paper/2305.06547/) (2023)
- [Backward Reachability Analysis of Neural Feedback Loops: Techniques for Linear and Nonlinear Systems](https://mlsanity.com/paper/2209.14076/) (2022)
- [Stable and Safe Reinforcement Learning via a Barrier-Lyapunov Actor-Critic Approach](https://mlsanity.com/paper/2304.04066/) (2023)
- [Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions](https://mlsanity.com/paper/2305.09793/) (2023)
- [Neural Koopman Lyapunov Control](https://mlsanity.com/paper/2201.05098/) (2022)
- [Variational Quantum Soft Actor-Critic for Robotic Arm Control](https://mlsanity.com/paper/2212.11681/) (2022)
- [Critic as Lyapunov function (CALF): a model-free, stability-ensuring agent](https://mlsanity.com/paper/2409.09869/) (2024)
- [Stabilizing reinforcement learning control: A modular framework for optimizing over all stable behavior](https://mlsanity.com/paper/2310.14098/) (2023)
- [Stable and Safe Human-aligned Reinforcement Learning through Neural Ordinary Differential Equations](https://mlsanity.com/paper/2401.13148/) (2024)
- [Transformer-Based Fault-Tolerant Control for Fixed-Wing UAVs Using Knowledge Distillation and In-Context Adaptation](https://mlsanity.com/paper/2411.02975/) (2024)
- [Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learning](https://mlsanity.com/paper/2203.12048/) (2022)
- [A Learning Approach for Joint Design of Event-triggered Control and Power-Efficient Resource Allocation](https://mlsanity.com/paper/2205.07070/) (2022)
- [Stochastic Reinforcement Learning with Stability Guarantees for Control of Unknown Nonlinear Systems](https://mlsanity.com/paper/2409.08382/) (2024)
- [Sample-Efficient Reinforcement Learning Controller for Deep Brain Stimulation in Parkinson's Disease](https://mlsanity.com/paper/2507.06326/) (2025)
- [Off Policy Lyapunov Stability in Reinforcement Learning](https://mlsanity.com/paper/2509.09863/) (2025)
- [Reinforcement Learning for Orientation Estimation Using Inertial Sensors with Performance Guarantee](https://mlsanity.com/paper/2103.02357/) (2021)
- [Robust Stability of Neural Network-controlled Nonlinear Systems with Parametric Variability](https://mlsanity.com/paper/2109.05710/) (2021)
- [A Policy Optimization Method Towards Optimal-time Stability](https://mlsanity.com/paper/2301.00521/) (2023)
- [A modular framework for stabilizing deep reinforcement learning control](https://mlsanity.com/paper/2304.03422/) (2023)
- [Stable Modular Control via Contraction Theory for Reinforcement Learning](https://mlsanity.com/paper/2311.03669/) (2023)
- [A novel agent with formal goal-reaching guarantees: an experimental study with a mobile robot](https://mlsanity.com/paper/2409.14867/) (2024)
- [Neural Lyapunov Function Approximation with Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2503.15629/) (2025)
- [Learning Verifiable Control Policies Using Relaxed Verification](https://mlsanity.com/paper/2504.16879/) (2025)
- [Vision-based Goal-Reaching Control for Mobile Robots Using a Hierarchical Learning Framework](https://mlsanity.com/paper/2601.00610/) (2026)
- [LC-SAC: Lyapunov-Constrained Soft Actor-Critic via Koopman Operator Theory for Trajectory Tracking and Stabilization](https://mlsanity.com/paper/2602.04132/) (2026)
- [ContractionPPO: Certified Reinforcement Learning via Differentiable Contraction Layers](https://mlsanity.com/paper/2603.19632/) (2026)
- [Accelerate Reinforcement Learning with PID Controllers in the Pendulum Simulations](https://mlsanity.com/paper/2210.00770/) (2022)
- [Efficient Exploration Using Extra Safety Budget in Constrained Policy Optimization](https://mlsanity.com/paper/2302.14339/) (2023)
- [Deep Bayesian Reinforcement Learning for Spacecraft Proximity Maneuvers and Docking](https://mlsanity.com/paper/2311.03680/) (2023)

## Abstract (arXiv)

Reinforcement Learning (RL) and its integration with deep learning have achieved impressive performance in various robotic control tasks, ranging from motion planning and navigation to end-to-end visual manipulation. However, stability is not guaranteed in model-free RL by solely using data. From a control-theoretic perspective, stability is the most important property for any control system, since it is closely related to safety, robustness, and reliability of robotic systems. In this paper, we propose an actor-critic RL framework for control which can guarantee closed-loop stability by employing the classic Lyapunov's method in control theory. First of all, a data-based stability theorem is proposed for stochastic nonlinear systems modeled by Markov decision process. Then we show that the stability condition could be exploited as the critic in the actor-critic RL to learn a controller/policy. At last, the effectiveness of our approach is evaluated on several well-known 3-dimensional robot control tasks and a synthetic biology gene network tracking task in three different popular physics simulation platforms. As an empirical evaluation on the advantage of stability, we show that the learned policies can enable the systems to recover to the equilibrium or way-points when interfered by uncertainties such as system parametric variations and external disturbances to a certain extent.