mlsanityData to 7 Oct 2026

A Lyapunov-based Approach to Safe Reinforcement Learning

arXiv 1805.07708 ↗First posted 20 May 2018cs.LG cs.AI stat.MLcited by 278

Not on the map. Why?

Abstract (arXiv)

In many real-world reinforcement learning (RL) problems, besides optimizing the main objective function, an agent must concurrently avoid violating a number of constraints. In particular, besides optimizing performance it is crucial to guarantee the safety of an agent during training as well as deployment (e.g. a robot should avoid taking actions - exploratory or not - which irrevocably harm its hardware). To incorporate safety in RL, we derive algorithms under the framework of constrained Markov decision problems (CMDPs), an extension of the standard Markov decision problems (MDPs) augmented with constraints on expected cumulative costs. Our approach hinges on a novel Lyapunov method. We define and present a method for constructing Lyapunov functions, which provide an effective way to guarantee the global safety of a behavior policy during training via a set of local, linear constraints. Leveraging these theoretical underpinnings, we show how to use the Lyapunov approach to systematically transform dynamic programming (DP) and RL algorithms into their safe counterparts. To illustrate their effectiveness, we evaluate these algorithms in several CMDP planning and decision-making tasks on a safety benchmark domain. Our results show that our proposed method significantly outperforms existing baselines in balancing constraint satisfaction and performance.

Key facts. Cited by 278 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 48; 2022: 44; 2023: 49; 2024: 48; 2025: 37; 2026: 52.

278 papers.

Sort:

By area:

  1. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  2. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  3. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  4. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 83
  5. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  6. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 63
  7. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 55
  8. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 54
  9. 2023State-wise Safe Reinforcement Learning: A Surveycited by 48
  10. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 47
  11. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 45
  12. 2021Safe Reinforcement Learning Using Robust Control Barrier Functionscited by 44
38 more
  1. 2022LyaNet: A Lyapunov Framework for Training Neural ODEscited by 43
  2. 2021Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challengescited by 40
  3. 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
  4. 2021Context-Aware Safe Reinforcement Learning for Non-Stationary Environmentscited by 33
  5. 2021Safe Reinforcement Learning with Linear Function Approximationcited by 33
  6. 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 32
  7. 2021Multi-Agent Constrained Policy Optimisationcited by 32
  8. 2022Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarkingcited by 31
  9. 2021Safe Driving via Expert Guided Policy Optimizationcited by 29
  10. 2022Safe Reinforcement Learning for Legged Locomotioncited by 29
  11. 2025Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimizationcited by 29
  12. 2022Towards Safe Reinforcement Learning with a Safety Editor Policycited by 28
  13. 2021GoSafe: Globally Optimal Safe Robot Learningcited by 27
  14. 2022MuZero with Self-competition for Rate Control in VP9 Video Compressioncited by 27
  15. 2022Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithmcited by 27
  16. 2023MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulationcited by 27
  17. 2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 26
  18. 2021Safe Pontryagin Differentiable Programmingcited by 25
  19. 2021Density Constrained Reinforcement Learningcited by 25
  20. 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPscited by 22
  21. 2022Safe Reinforcement Learning Using Black-Box Reachability Analysiscited by 22
  22. 2022Learning Control Policies for Stochastic Systems with Reach-avoid Guaranteescited by 21
  23. 2024Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulationcited by 21
  24. 2022Online Shielding for Reinforcement Learningcited by 19
  25. 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 19
  26. 2023RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorizationcited by 19
  27. 2021Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimizationcited by 18
  28. 2021Faster Algorithm and Sharper Analysis for Constrained Markov Decision Processcited by 18
  29. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 18
  30. 2021Safe Continuous Control with Constrained Model-Based Policy Optimizationcited by 17
  31. 2021A Provably-Efficient Model-Free Algorithm for Constrained Markov Decision Processescited by 17
  32. 2022GoSafeOpt: Scalable Safe Exploration for Global Optimization of Dynamical Systemscited by 17
  33. 2022Lexicographic Multi-Objective Reinforcement Learningcited by 17
  34. 2021Almost Surely Stable Deep Dynamicscited by 16
  35. 2022CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learningcited by 16
  36. 2023AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learningcited by 16
  37. 2023Neural Operators of Backstepping Controller and Observer Gain Functions for Reaction-Diffusion PDEscited by 16
  38. 2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learningcited by 16

50 most cited of 278 All 278 (JSON)