# A framework for online, stabilizing reinforcement learning

arXiv: https://arxiv.org/abs/2207.08730 · HTML: https://mlsanity.com/paper/2207.08730/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 47 references that are in the map.
- First posted on arXiv 18 Jul 2022.

## Builds on (most cited first, 17 of 17)

- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [Trial without Error: Towards Safe Reinforcement Learning via Human Intervention](https://mlsanity.com/paper/1707.05173/) (2017)
- [Safe Reinforcement Learning Using Robust MPC](https://mlsanity.com/paper/1906.04005/) (2019)
- [Practical Reinforcement Learning of Stabilizing Economic MPC](https://mlsanity.com/paper/1904.04614/) (2019)
- [Deep Reinforcement learning for real autonomous mobile robot navigation in indoor environments](https://mlsanity.com/paper/2005.13857/) (2020)
- [Concurrent learning-based approximate optimal regulation](https://mlsanity.com/paper/1304.3477/) (2013)
- [Global Adaptive Dynamic Programming for Continuous-Time Nonlinear Systems](https://mlsanity.com/paper/1401.0020/) (2013)
- [Practical sample-and-hold stabilization of nonlinear systems under approximate optimizers](https://mlsanity.com/paper/1803.02203/) (2018)
- [Recursively Feasible Stochastic Model Predictive Control using Indirect Feedback](https://mlsanity.com/paper/1812.06860/) (2018)
- [On inf-convolution-based robust practical stabilization under computational uncertainty](https://mlsanity.com/paper/2102.04309/) (2021)
- [Model predictive control with stage cost shaping inspired by reinforcement learning](https://mlsanity.com/paper/1906.02580/) (2019)
- [A reinforcement learning method with closed-loop stability guarantee](https://mlsanity.com/paper/2006.14034/) (2020)
- [A note on stabilizing reinforcement learning](https://mlsanity.com/paper/2111.12316/) (2021)
- [On stochastic stabilization via non-smooth control Lyapunov functions](https://mlsanity.com/paper/2205.13409/) (2022)
- [On stochastic stabilization of sampled systems](https://mlsanity.com/paper/2105.07152/) (2021)

## Abstract (arXiv)

Online reinforcement learning is concerned with training an agent on-the-fly via dynamic interaction with the environment. Here, due to the specifics of the application, it is not generally possible to perform long pre-training, as it is commonly done in off-line, model-free approaches, which are akin to dynamic programming. Such applications may be found more frequently in industry, rather than in pure digital fields, such as cloud services, video games, database management, etc., where reinforcement learning has been demonstrating success. Online reinforcement learning, in contrast, is more akin to classical control, which utilizes some model knowledge about the environment. Stability of the closed-loop (agent plus the environment) is a major challenge for such online approaches. In this paper, we tackle this problem by a special fusion of online reinforcement learning with elements of classical control, namely, based on the Lyapunov theory of stability. The idea is to start the agent at once, without pre-training, and learn approximately optimal policy under specially designed constraints, which guarantee stability. The resulting approach was tested in an extensive experimental study with a mobile robot. A nominal parking controller was used as a baseline. It was observed that the suggested agent could always successfully park the robot, while significantly improving the cost. While many approaches may be exploited for mobile robot control, we suggest that the experiments showed the promising potential of online reinforcement learning agents based on Lyapunov-like constraints. The presented methodology may be utilized in safety-critical, industrial applications where stability is necessary.