mlsanityData to 14 Oct 2026

Off Policy Lyapunov Stability in Reinforcement Learning

arXiv 2509.09863 ↗First posted 11 Sep 2025eess.SY cs.LG cs.ROcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Traditional reinforcement learning lacks the ability to provide stability guarantees. More recent algorithms learn Lyapunov functions alongside the control policies to ensure stable learning. However, the current self-learned Lyapunov functions are sample inefficient due to their on-policy nature. This paper introduces a method for learning Lyapunov functions off-policy and incorporates the proposed off-policy Lyapunov function into the Soft Actor Critic and Proximal Policy Optimization algorithms to provide them with a data efficient stability certificate. Simulations of an inverted pendulum and a quadrotor illustrate the improved performance of the two algorithms when endowed with the proposed off-policy Lyapunov function.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 26 references that are in the map. First posted on arXiv 11 Sep 2025.

Builds on

20 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Critic as Lyapunov function (CALF): a model-free, stability-ensuring agent15 Sep

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  4. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  5. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  6. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 289
  7. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 176
  8. 2022Safe Control with Learned Certificates: A Survey of Neural Lyapunov, Barrier, and Contraction methodscited by 151
  9. 2020Neural Lyapunov Controlcited by 142
  10. 2017Control of a Quadrotor with Reinforcement Learningcited by 92
  11. 2020Learning Certified Control using Contraction Metriccited by 49
  12. 2022A Benchmark Comparison of Learned Control Policies for Agile Quadrotor Flightcited by 47
7 more
  1. 2020Actor-Critic Reinforcement Learning for Control with Stability Guaranteecited by 45
  2. 2021Stabilizing Neural Control Using Self-Learned Almost Lyapunov Criticscited by 35
  3. 2023Physics-Informed Neural Network Lyapunov Functions: PDE Characterization, Learning, and Verificationcited by 20
  4. 2022Refining Control Barrier Functions through Hamilton-Jacobi Reachabilitycited by 15
  5. 2023Stable and Safe Reinforcement Learning via a Barrier-Lyapunov Actor-Critic Approachcited by 13
  6. 2023Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functionscited by 13
  7. 2022Model-free Neural Lyapunov Control for Safe Robot Navigationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026LC-SAC: Lyapunov-Constrained Soft Actor-Critic via Koopman Operator Theory for Trajectory Tracking and Stabilizationcited by 1
  2. 2026When Does External Guidance Help LLM Reasoning? A Bias-Variance Theory of Guidance-Augmented GRPOcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Certifying Stability of Reinforcement Learning Policies using Generalized Lyapunov Functions
  2. 2025Neural Lyapunov Function Approximation with Self-Supervised Reinforcement Learning
  3. 2025MSACL: Multi-Step Actor-Critic Learning with Lyapunov Certificates for Exponentially Stabilizing Control
  4. 2021Offline RL Without Off-Policy Evaluation
  5. 2023Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learning
  6. 2026Reinforcement Learning for Control with Probabilistic Stability Guarantee: A Finite-Sample Approach
  7. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
  8. 2024Optimization Solution Functions as Deterministic Policies for Offline Reinforcement Learning