# Critic as Lyapunov function (CALF): a model-free, stability-ensuring agent

arXiv: https://arxiv.org/abs/2409.09869 · HTML: https://mlsanity.com/paper/2409.09869/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 24 of its 56 references that are in the map.
- First posted on arXiv 15 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 24 of 24)

- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones](https://mlsanity.com/paper/2010.15920/) (2020)
- [Learning-based Model Predictive Control for Safe Exploration](https://mlsanity.com/paper/1803.08287/) (2018)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [Trial without Error: Towards Safe Reinforcement Learning via Human Intervention](https://mlsanity.com/paper/1707.05173/) (2017)
- [Safe Reinforcement Learning Using Robust MPC](https://mlsanity.com/paper/1906.04005/) (2019)
- [Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions](https://mlsanity.com/paper/2004.07584/) (2020)
- [Safe Reinforcement Learning on Autonomous Vehicles](https://mlsanity.com/paper/1910.00399/) (2019)
- [Actor-Critic Reinforcement Learning for Control with Stability Guarantee](https://mlsanity.com/paper/2004.14288/) (2020)
- [Learning Human Objectives by Evaluating Hypothetical Behavior](https://mlsanity.com/paper/1912.05652/) (2019)
- [Stabilizing Neural Control Using Self-Learned Almost Lyapunov Critics](https://mlsanity.com/paper/2107.04989/) (2021)
- [Deep Value Model Predictive Control](https://mlsanity.com/paper/1910.03358/) (2019)
- [Practical Reinforcement Learning of Stabilizing Economic MPC](https://mlsanity.com/paper/1904.04614/) (2019)
- [Infinite-Horizon Differentiable Model Predictive Control](https://mlsanity.com/paper/2001.02244/) (2020)
- [Practical Reinforcement Learning For MPC: Learning from sparse objectives in under an hour on a real robot](https://mlsanity.com/paper/2003.03200/) (2020)
- [Fixed-Horizon Temporal Difference Methods for Stable Reinforcement Learning](https://mlsanity.com/paper/1909.03906/) (2019)
- [Deductive Stability Proofs for Ordinary Differential Equations](https://mlsanity.com/paper/2010.13096/) (2020)
- [Transferring Multiple Policies to Hotstart Reinforcement Learning in an Air Compressor Management Problem](https://mlsanity.com/paper/2301.12820/) (2023)

## Built on by (most cited first, 4 of 4)

- [Off Policy Lyapunov Stability in Reinforcement Learning](https://mlsanity.com/paper/2509.09863/) (2025)
- [A novel agent with formal goal-reaching guarantees: an experimental study with a mobile robot](https://mlsanity.com/paper/2409.14867/) (2024)
- [Multi-CALF: A Policy Combination Approach with Statistical Guarantees](https://mlsanity.com/paper/2505.12350/) (2025)
- [A universal policy wrapper with guarantees](https://mlsanity.com/paper/2505.12354/) (2025)

## Abstract (arXiv)

This work presents and showcases a novel reinforcement learning agent called Critic As Lyapunov Function (CALF) which is model-free and ensures online environment, in other words, dynamical system stabilization. Online means that in each learning episode, the said environment is stabilized. This, as demonstrated in a case study with a mobile robot simulator, greatly improves the overall learning performance. The base actor-critic scheme of CALF is analogous to SARSA. The latter did not show any success in reaching the target in our studies. However, a modified version thereof, called SARSA-m here, did succeed in some learning scenarios. Still, CALF greatly outperformed the said approach. CALF was also demonstrated to improve a nominal stabilizer provided to it. In summary, the presented agent may be considered a viable approach to fusing classical control with reinforcement learning. Its concurrent approaches are mostly either offline or model-based, like, for instance, those that fuse model-predictive control into the agent.