# Safe Deep Model-Based Reinforcement Learning with Lyapunov Functions

arXiv: https://arxiv.org/abs/2405.16184 · HTML: https://mlsanity.com/paper/2405.16184/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 25 May 2024.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 25 of 25)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects](https://mlsanity.com/paper/2205.04382/) (2022)
- [Learning Predictive Representations for Deformable Objects Using Contrastive Estimation](https://mlsanity.com/paper/2003.05436/) (2020)
- [TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/2211.09325/) (2022)
- [Personalization of End-to-end Speech Recognition On Mobile Devices For Named Entities](https://mlsanity.com/paper/1912.09251/) (2019)
- [Learning Model Predictive Control for iterative tasks. A Data-Driven Control Framework](https://mlsanity.com/paper/1609.01387/) (2016)
- [FlowBot++: Learning Generalized Articulated Objects Manipulation via Articulation Projection](https://mlsanity.com/paper/2306.12893/) (2023)
- [One-Shot Learning of Manipulation Skills with Online Dynamics Adaptation and Neural Network Priors](https://mlsanity.com/paper/1509.06841/) (2015)
- [DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classification](https://mlsanity.com/paper/2305.15957/) (2023)
- [Robots of the Lost Arc: Self-Supervised Learning to Dynamically Manipulate Fixed-Endpoint Cables](https://mlsanity.com/paper/2011.04840/) (2020)
- [Orienting Novel 3D Objects Using Self-Supervised Learning of Rotation Transforms](https://mlsanity.com/paper/2105.14246/) (2021)
- [6-DoF Grasp Planning using Fast 3D Reconstruction and Grasp Quality CNN](https://mlsanity.com/paper/2009.08618/) (2020)
- [ART/ATK: A research platform for assessing and mitigating the sim-to-real gap in robotics and autonomous vehicle engineering](https://mlsanity.com/paper/2211.04886/) (2022)
- [APLA: Additional Perturbation for Latent Noise with Adversarial Training Enables Consistency](https://mlsanity.com/paper/2308.12605/) (2023)
- [ABC-LMPC: Safe Sample-Based Learning MPC for Stochastic Nonlinear Dynamical Systems with Adjustable Boundary Conditions](https://mlsanity.com/paper/2003.01410/) (2020)
- [Sample-Based Learning Model Predictive Control for Linear Uncertain Systems](https://mlsanity.com/paper/1904.06432/) (2019)
- [Sampling-based verification of Lyapunov's inequality for piecewise continuous nonlinear systems](https://mlsanity.com/paper/1609.00302/) (2016)
- [Safe Interactive Model-Based Learning](https://mlsanity.com/paper/1911.06556/) (2019)

## Built on by (most cited first, 2 of 2)

- [WMAttack: Automated Attack Search for Adversarial Evaluation of World-Model Agents](https://mlsanity.com/paper/2605.23220/) (2026)
- [H2OFlow: Grounding Human-Object Affordances with 3D Generative Models and Dense Diffused Flows](https://mlsanity.com/paper/2510.21769/) (2025)

## Abstract (arXiv)

Model-based Reinforcement Learning (MBRL) has shown many desirable properties for intelligent control tasks. However, satisfying safety and stability constraints during training and rollout remains an open question. We propose a new Model-based RL framework to enable efficient policy learning with unknown dynamics based on learning model predictive control (LMPC) framework with mathematically provable guarantees of stability. We introduce and explore a novel method for adding safety constraints for model-based RL during training and policy learning. The new stability-augmented framework consists of a neural-network-based learner that learns to construct a Lyapunov function, and a model-based RL agent to consistently complete the tasks while satisfying user-specified constraints given only sub-optimal demonstrations and sparse-cost feedback. We demonstrate the capability of the proposed framework through simulated experiments.