# Lyapunov Density Models: Constraining Distribution Shift in Learning-Based Control

arXiv: https://arxiv.org/abs/2206.10524 · HTML: https://mlsanity.com/paper/2206.10524/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 21 Jun 2022.

## Builds on (most cited first, 27 of 27)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [Neural Spline Flows](https://mlsanity.com/paper/1906.04032/) (2019)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Control Barrier Function Based Quadratic Programs for Safety Critical Systems](https://mlsanity.com/paper/1609.06408/) (2016)
- [Hamilton-Jacobi Reachability: A Brief Overview and Recent Advances](https://mlsanity.com/paper/1709.07523/) (2017)
- [Safe Policy Improvement with Baseline Bootstrapping](https://mlsanity.com/paper/1712.06924/) (2017)
- [Neural Lyapunov Control](https://mlsanity.com/paper/2005.00611/) (2020)
- [Reinforcement Learning for Robust Parameterized Locomotion Control of Bipedal Robots](https://mlsanity.com/paper/2103.14295/) (2021)
- [Gaussian Processes for Data-Efficient Learning in Robotics and Control](https://mlsanity.com/paper/1502.02860/) (2015)
- [Learning Control Barrier Functions from Expert Demonstrations](https://mlsanity.com/paper/2004.03315/) (2020)
- [The Lyapunov Neural Network: Adaptive Stability Certification for Safe Learning of Dynamical Systems](https://mlsanity.com/paper/1808.00924/) (2018)
- [Lyapunov-stable neural-network control](https://mlsanity.com/paper/2109.14152/) (2021)
- [On the Sample Complexity of Stability Constrained Imitation Learning](https://mlsanity.com/paper/2102.09161/) (2021)
- [Guaranteeing Safety of Learned Perception Modules via Measurement-Robust Control Barrier Functions](https://mlsanity.com/paper/2010.16001/) (2020)
- [Robustness to Out-of-Distribution Inputs via Task-Aware Generative Uncertainty](https://mlsanity.com/paper/1812.10687/) (2018)
- [Learning Dynamical Systems using Local Stability Priors](https://mlsanity.com/paper/2008.10053/) (2020)

## Built on by (most cited first, 28 of 28)

- [Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failures](https://mlsanity.com/paper/2505.00779/) (2025)
- [Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificate](https://mlsanity.com/paper/2210.07553/) (2022)
- [Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching](https://mlsanity.com/paper/2306.14079/) (2023)
- [In-Distribution Barrier Functions: Self-Supervised Policy Filters that Avoid Out-of-Distribution States](https://mlsanity.com/paper/2301.12012/) (2023)
- [SAFE-GIL: SAFEty Guided Imitation Learning for Robotic Systems](https://mlsanity.com/paper/2404.05249/) (2024)
- [A Survey of Demonstration Learning](https://mlsanity.com/paper/2303.11191/) (2023)
- [Predictable MDP Abstraction for Unsupervised Model-Based RL](https://mlsanity.com/paper/2302.03921/) (2023)
- [Elastic Motion Policy: An Adaptive Dynamical System for Robust and Efficient One-Shot Imitation Learning](https://mlsanity.com/paper/2503.08029/) (2025)
- [Learning Conservative Neural Control Barrier Functions from Offline Data](https://mlsanity.com/paper/2505.00908/) (2025)
- [Evaluating Defences against Unsafe Feedback in RLHF](https://mlsanity.com/paper/2409.12914/) (2024)
- [Safety Representations for Safer Policy Learning](https://mlsanity.com/paper/2502.20341/) (2025)
- [Pixels to Proofs: Probabilistically-Safe Latent World Model Control via Parallel Conformal Robust MPC](https://mlsanity.com/paper/2606.15594/) (2026)
- [Model Monitoring and Robustness of In-Use Machine Learning Models: Quantifying Data Distribution Shifts Using Population Stability Index](https://mlsanity.com/paper/2302.00775/) (2023)
- [A Conservative Approach for Few-Shot Transfer in Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2312.15474/) (2023)
- [Safety, Security, and Cognitive Risks in World Models](https://mlsanity.com/paper/2604.01346/) (2026)
- [SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situations](https://mlsanity.com/paper/2311.03651/) (2023)
- [Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows](https://mlsanity.com/paper/2405.03892/) (2024)
- [SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration](https://mlsanity.com/paper/2606.10228/) (2026)
- [Sim-to-Real Transfer of Adaptive Control Parameters for AUV Stabilization under Current Disturbance](https://mlsanity.com/paper/2310.11075/) (2023)
- [LaMOuR: Leveraging Language Models for Out-of-Distribution Recovery in Reinforcement Learning](https://mlsanity.com/paper/2503.17125/) (2025)
- [Distributionally Robust Imitation Learning: Layered Control Architecture for Certifiable Autonomy](https://mlsanity.com/paper/2512.17899/) (2025)
- [DEFENDER: DTW-Based Episode Filtering Using Demonstrations for Enhancing RL Safety](https://mlsanity.com/paper/2305.04727/) (2023)
- [Dissipative Imitation Learning for Discrete Dynamic Output Feedback Control with Sparse Data Sets](https://mlsanity.com/paper/2309.06658/) (2023)
- [Solving Parameter-Robust Avoid Problems with Unknown Feasibility using Reinforcement Learning](https://mlsanity.com/paper/2602.15817/) (2026)
- [Lyapunov-Guided Self-Alignment: Test-Time Adaptation for Offline Safe Reinforcement Learning](https://mlsanity.com/paper/2604.26516/) (2026)
- [Intrinsic Vicarious Conditioning for Deep Reinforcement Learning](https://mlsanity.com/paper/2605.12224/) (2026)
- [SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions](https://mlsanity.com/paper/2607.08925/) (2026)
- [Learning-Based Motion Planning for Dynamic Environments: From Foundational Algorithms to Emerging Paradigms](https://mlsanity.com/paper/2608.00625/) (2026)

## Abstract (arXiv)

Learned models and policies can generalize effectively when evaluated within the distribution of the training data, but can produce unpredictable and erroneous outputs on out-of-distribution inputs. In order to avoid distribution shift when deploying learning-based control algorithms, we seek a mechanism to constrain the agent to states and actions that resemble those that it was trained on. In control theory, Lyapunov stability and control-invariant sets allow us to make guarantees about controllers that stabilize the system around specific states, while in machine learning, density models allow us to estimate the training data distribution. Can we combine these two concepts, producing learning-based control algorithms that constrain the system to in-distribution states using only in-distribution actions? In this work, we propose to do this by combining concepts from Lyapunov stability and density estimation, introducing Lyapunov density models: a generalization of control Lyapunov functions and density models that provides guarantees on an agent's ability to stay in-distribution over its entire trajectory.