mlsanityData to 7 Oct 2026

Lyapunov-based Safe Policy Optimization for Continuous Control

arXiv 1901.10031 ↗First posted 28 Jan 2019cs.LG cs.AI stat.MLcited by 140

Not on the map. Why?

Abstract (arXiv)

We study continuous action reinforcement learning problems in which it is crucial that the agent interacts with the environment only through safe policies, i.e.,~policies that do not take the agent to undesirable situations. We formulate these problems as constrained Markov decision processes (CMDPs) and present safe policy optimization algorithms that are based on a Lyapunov approach to solve them. Our algorithms can use any standard policy gradient (PG) method, such as deep deterministic policy gradient (DDPG) or proximal policy optimization (PPO), to train a neural network policy, while guaranteeing near-constraint satisfaction for every policy update by projecting either the policy parameter or the action onto the set of feasible solutions induced by the state-dependent linearized Lyapunov constraints. Compared to the existing constrained PG algorithms, ours are more data efficient as they are able to utilize both on-policy and off-policy data. Moreover, our action-projection algorithm often leads to less conservative policy updates and allows for natural integration into an end-to-end PG training pipeline. We evaluate our algorithms and compare them with the state-of-the-art baselines on several simulated (MuJoCo) tasks, as well as a real-world indoor robot navigation problem, demonstrating their effectiveness in terms of balancing performance and constraint satisfaction. Videos of the experiments can be found in the following link: https://drive.google.com/file/d/1pzuzFqWIE710bE2U6DmS59AfRzqK2Kek/view?usp=sharing.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 25; 2023: 24; 2024: 31; 2025: 13; 2026: 22.

140 papers.

Sort:

By area:

  1. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  2. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  3. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  4. 2023Motion Planning for Autonomous Driving: The State of the Art and Future Perspectivescited by 112
  5. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  6. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 63
  7. 2022Constrained Policy Optimization via Bayesian World Modelscited by 55
  8. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  9. 2023State-wise Safe Reinforcement Learning: A Surveycited by 48
  10. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 47
  11. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 45
  12. 2021Safe Reinforcement Learning Using Robust Control Barrier Functionscited by 44
38 more
  1. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  2. 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
  3. 2021Context-Aware Safe Reinforcement Learning for Non-Stationary Environmentscited by 33
  4. 2023Iterative Reachability Estimation for Safe Reinforcement Learningcited by 33
  5. 2021Multi-Agent Constrained Policy Optimisationcited by 32
  6. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  7. 2021Direct Behavior Specification via Constrained Reinforcement Learningcited by 30
  8. 2022Safe Reinforcement Learning for Legged Locomotioncited by 29
  9. 2022Towards Safe Reinforcement Learning with a Safety Editor Policycited by 28
  10. 2022Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithmcited by 27
  11. 2021Safe Pontryagin Differentiable Programmingcited by 25
  12. 2021Differentiable Trust Region Layers for Deep Reinforcement Learningcited by 24
  13. 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 22
  14. 2024Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulationcited by 21
  15. 2021Faster Algorithm and Sharper Analysis for Constrained Markov Decision Processcited by 18
  16. 2022Benchmarking Constraint Inference in Inverse Reinforcement Learningcited by 18
  17. 2022Safe Reinforcement Learning of Dynamic High-Dimensional Robotic Tasks: Navigation, Manipulation, Interactioncited by 15
  18. 2022Benchmarking Reinforcement Learning Techniques for Autonomous Navigationcited by 15
  19. 2021Safe Autonomous Racing via Approximate Reachability on Ego-visioncited by 13
  20. 2021Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learningcited by 13
  21. 2022Policy Optimization with Advantage Regularization for Long-Term Fairness in Decision Systemscited by 13
  22. 2023A Human-Centered Safe Robot Reinforcement Learning Framework with Interactive Behaviorscited by 13
  23. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  24. 2021Lyapunov Barrier Policy Optimizationcited by 12
  25. 2022ROI-Constrained Bidding via Curriculum-Guided Bayesian Reinforcement Learningcited by 12
  26. 2024Constrained Reinforcement Learning with Smoothed Log Barrier Functioncited by 12
  27. 2024Safe Reinforcement Learning on the Constraint Manifold: Theory and Applicationscited by 12
  28. 2024Hamilton-Jacobi Reachability in Reinforcement Learning: A Surveycited by 11
  29. 2022Robotic Table Wiping via Reinforcement Learning and Whole-body Trajectory Optimizationcited by 10
  30. 2023Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control with Action Constraintscited by 10
  31. 2023A safe exploration approach to constrained Markov decision processescited by 10
  32. 2024Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learningcited by 10
  33. 2022Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probabilitycited by 9
  34. 2023Safe Deep Reinforcement Learning by Verifying Task-Level Propertiescited by 9
  35. 2023Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functionscited by 9
  36. 2023State-wise Constrained Policy Optimizationcited by 9
  37. 2024Safe and Balanced: A Framework for Constrained Multi-Objective Reinforcement Learningcited by 9
  38. 2024Embedding Safety into RL: A New Take on Trust Region Methodscited by 9

50 most cited of 140 All 140 (JSON)