# Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning

arXiv: https://arxiv.org/abs/2208.06721 · HTML: https://mlsanity.com/paper/2208.06721/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2022.

## Built on by (most cited first, 28 of 28)

- [SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning](https://mlsanity.com/paper/2401.16013/) (2024)
- [Reinforcement Learning for Versatile, Dynamic, and Robust Bipedal Locomotion Control](https://mlsanity.com/paper/2401.16889/) (2024)
- [Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuning](https://mlsanity.com/paper/2502.02705/) (2025)
- [Learning to Walk from Three Minutes of Real-World Data with Semi-structured Dynamics Models](https://mlsanity.com/paper/2410.09163/) (2024)
- [Benchmarking Potential Based Rewards for Learning Humanoid Locomotion](https://mlsanity.com/paper/2307.10142/) (2023)
- [Physics-Regulated Deep Reinforcement Learning: Invariant Embeddings](https://mlsanity.com/paper/2305.16614/) (2023)
- [Emergent Dexterity via Diverse Resets and Large-Scale Reinforcement Learning](https://mlsanity.com/paper/2603.15789/) (2026)
- [Value Functions are Control Barrier Functions: Verification of Safe Policies using Control Theory](https://mlsanity.com/paper/2306.04026/) (2023)
- [Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptation](https://mlsanity.com/paper/2603.15759/) (2026)
- [CLF-RL: Control Lyapunov Function Guided Reinforcement Learning](https://mlsanity.com/paper/2508.09354/) (2025)
- [A Review On Safe Reinforcement Learning Using Lyapunov and Barrier Functions](https://mlsanity.com/paper/2508.09128/) (2025)
- [Physical Deep Reinforcement Learning Towards Safety Guarantee](https://mlsanity.com/paper/2303.16860/) (2023)
- [Decomposing Control Lyapunov Functions for Efficient Reinforcement Learning](https://mlsanity.com/paper/2403.12210/) (2024)
- [OHP-RL: Online Human Preference as Guidance in Reinforcement Learning for Robot Manipulation](https://mlsanity.com/paper/2605.15971/) (2026)
- [Distributionally Robust Policy and Lyapunov-Certificate Learning](https://mlsanity.com/paper/2404.03017/) (2024)
- [Simplex-enabled Safe Continual Learning Machine](https://mlsanity.com/paper/2409.05898/) (2024)
- [Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrations](https://mlsanity.com/paper/2412.01114/) (2024)
- [Gait Switching and Enhanced Stabilization of Walking Robots with Deep Learning-based Reachability: A Case Study on Two-link Walker](https://mlsanity.com/paper/2409.16301/) (2024)
- [Control Synthesis with Reinforcement Learning: A Modeling Perspective](https://mlsanity.com/paper/2510.25063/) (2025)
- [MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins](https://mlsanity.com/paper/2606.26392/) (2026)
- [Safe and Robust Neural Policy Learning with Statistical Verification for Sim-to-Real Deployment in Robotics](https://mlsanity.com/paper/2608.06481/) (2026)
- [Learning to Stabilize Online Reinforcement Learning in Unbounded State Spaces](https://mlsanity.com/paper/2306.01896/) (2023)
- [Enabling Efficient, Reliable Real-World Reinforcement Learning with Approximate Physics-Based Models](https://mlsanity.com/paper/2307.08168/) (2023)
- [Physics-model-guided Worst-case Sampling for Safe Reinforcement Learning](https://mlsanity.com/paper/2412.13224/) (2024)
- [Real-DRL: Teach and Learn in Reality](https://mlsanity.com/paper/2511.00112/) (2025)
- [Stability of Control Lyapunov Function Guided Reinforcement Learning](https://mlsanity.com/paper/2605.01978/) (2026)
- [Zero-shot Transfer of Reinforcement Learning Control Policies for the Swing-Up and Stabilization of a Cart-Pole System](https://mlsanity.com/paper/2606.22145/) (2026)
- [Safe Online Learning via Smooth Safety-Structured Policy Composition](https://mlsanity.com/paper/2606.31320/) (2026)

## Abstract (arXiv)

Recent advances in the reinforcement learning (RL) literature have enabled roboticists to automatically train complex policies in simulated environments. However, due to the poor sample complexity of these methods, solving RL problems using real-world data remains a challenging problem. This paper introduces a novel cost-shaping method which aims to reduce the number of samples needed to learn a stabilizing controller. The method adds a term involving a Control Lyapunov Function (CLF) -- an `energy-like' function from the model-based control literature -- to typical cost formulations. Theoretical results demonstrate the new costs lead to stabilizing controllers when smaller discount factors are used, which is well-known to reduce sample complexity. Moreover, the addition of the CLF term `robustifies' the search for a stabilizing controller by ensuring that even highly sub-optimal polices will stabilize the system. We demonstrate our approach with two hardware examples where we learn stabilizing controllers for a cartpole and an A1 quadruped with only seconds and a few minutes of fine-tuning data, respectively. Furthermore, simulation benchmark studies show that obtaining stabilizing policies by optimizing our proposed costs requires orders of magnitude less data compared to standard cost designs.