# A novel agent with formal goal-reaching guarantees: an experimental study with a mobile robot

arXiv: https://arxiv.org/abs/2409.14867 · HTML: https://mlsanity.com/paper/2409.14867/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 21 of 21)

- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Projection-Based Constrained Policy Optimization](https://mlsanity.com/paper/2010.03152/) (2020)
- [Lyapunov-based Safe Policy Optimization for Continuous Control](https://mlsanity.com/paper/1901.10031/) (2019)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [Safe Reinforcement Learning Using Robust MPC](https://mlsanity.com/paper/1906.04005/) (2019)
- [Safe Reinforcement Learning Using Robust Control Barrier Functions](https://mlsanity.com/paper/2110.05415/) (2021)
- [Actor-Critic Reinforcement Learning for Control with Stability Guarantee](https://mlsanity.com/paper/2004.14288/) (2020)
- [Learning Human Objectives by Evaluating Hypothetical Behavior](https://mlsanity.com/paper/1912.05652/) (2019)
- [Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimization](https://mlsanity.com/paper/2003.01303/) (2020)
- [Stabilizing Neural Control Using Self-Learned Almost Lyapunov Critics](https://mlsanity.com/paper/2107.04989/) (2021)
- [Practical Reinforcement Learning of Stabilizing Economic MPC](https://mlsanity.com/paper/1904.04614/) (2019)
- [Infinite-Horizon Differentiable Model Predictive Control](https://mlsanity.com/paper/2001.02244/) (2020)
- [Practical Reinforcement Learning For MPC: Learning from sparse objectives in under an hour on a real robot](https://mlsanity.com/paper/2003.03200/) (2020)
- [Fixed-Horizon Temporal Difference Methods for Stable Reinforcement Learning](https://mlsanity.com/paper/1909.03906/) (2019)
- [Safe reinforcement learning for probabilistic reachability and safety specifications: A Lyapunov-based approach](https://mlsanity.com/paper/2002.10126/) (2020)
- [Model-free Neural Lyapunov Control for Safe Robot Navigation](https://mlsanity.com/paper/2203.01190/) (2022)
- [Critic as Lyapunov function (CALF): a model-free, stability-ensuring agent](https://mlsanity.com/paper/2409.09869/) (2024)
- [Deductive Stability Proofs for Ordinary Differential Equations](https://mlsanity.com/paper/2010.13096/) (2020)
- [A Multiplicative Value Function for Safe and Efficient Reinforcement Learning](https://mlsanity.com/paper/2303.04118/) (2023)

## Built on by (most cited first, 1 of 1)

- [Vision-based Goal-Reaching Control for Mobile Robots Using a Hierarchical Learning Framework](https://mlsanity.com/paper/2601.00610/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) has been shown to be effective and convenient for a number of tasks in robotics. However, it requires the exploration of a sufficiently large number of state-action pairs, many of which may be unsafe or unimportant. For instance, online model-free learning can be hazardous and inefficient in the absence of guarantees that a certain set of desired states will be reached during an episode. An increasingly common approach to address safety involves the addition of a shielding system that constrains the RL actions to a safe set of actions. In turn, a difficulty for such frameworks is how to effectively couple RL with the shielding system to make sure the exploration is not excessively restricted. This work presents a novel safe model-free RL agent called Critic As Lyapunov Function (CALF) and showcases how CALF can be used to improve upon control baselines in robotics in an efficient and convenient fashion while ensuring guarantees of stable goal reaching. The latter is a crucial part of safety, as seen generally. With CALF all state-action pairs remain explorable and yet reaching of desired goal states is formally guaranteed. Formal analysis is provided that shows the goal stabilization-ensuring properties of CALF and a set of real-world and numerical experiments with a non-holonomic wheeled mobile robot (WMR) TurtleBot3 Burger confirmed the superiority of CALF over such a well-established RL agent as proximal policy optimization (PPO), and a modified version of SARSA in a few-episode setting in terms of attained total cost.