# Stabilizing reinforcement learning control: A modular framework for optimizing over all stable behavior

arXiv: https://arxiv.org/abs/2310.14098 · HTML: https://mlsanity.com/paper/2310.14098/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 21 Oct 2023.

## Builds on (most cited first, 18 of 18)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Actor-Critic Reinforcement Learning for Control with Stability Guarantee](https://mlsanity.com/paper/2004.14288/) (2020)
- [Stabilizing Neural Control Using Self-Learned Almost Lyapunov Critics](https://mlsanity.com/paper/2107.04989/) (2021)
- [Stability-certified reinforcement learning: A control-theoretic perspective](https://mlsanity.com/paper/1810.11505/) (2018)
- [Stabilizing Dynamical Systems via Policy Gradient Methods](https://mlsanity.com/paper/2110.06418/) (2021)
- [Deep Reinforcement Learning with Shallow Controllers: An Experimental Application to PID Tuning](https://mlsanity.com/paper/2111.07171/) (2021)
- [Reinforcement Learning with Fast Stabilization in Linear Dynamical Systems](https://mlsanity.com/paper/2007.12291/) (2020)
- [Almost Surely Stable Deep Dynamics](https://mlsanity.com/paper/2103.14722/) (2021)
- [Recurrent Neural Network Controllers Synthesis with Stability Guarantees for Partially Observed Systems](https://mlsanity.com/paper/2109.03861/) (2021)
- [Learning for MPC with Stability & Safety Guarantees](https://mlsanity.com/paper/2012.07369/) (2020)
- [Learning over All Stabilizing Nonlinear Controllers for a Partially-Observed Linear System](https://mlsanity.com/paper/2112.04219/) (2021)
- [A trajectory-based framework for data-driven system analysis and control](https://mlsanity.com/paper/1903.10723/) (2019)
- [MathOptInterface: a data structure for mathematical optimization problems](https://mlsanity.com/paper/2002.03447/) (2020)
- [Willems' Fundamental Lemma for State-space Systems and its Extension to Multiple Datasets](https://mlsanity.com/paper/2002.01023/) (2020)
- [Approximating the nearest stable discrete-time system](https://mlsanity.com/paper/1802.08033/) (2018)
- [An Input-Output Parametrization of Stabilizing Controllers: amidst Youla and System Level Synthesis](https://mlsanity.com/paper/1903.03828/) (2019)
- [A modular framework for stabilizing deep reinforcement learning control](https://mlsanity.com/paper/2304.03422/) (2023)

## Built on by (most cited first, 3 of 3)

- [A view on learning robust goal-conditioned value functions: Interplay between RL and MPC](https://mlsanity.com/paper/2502.06996/) (2025)
- [Synthesizing Neural Network Controllers with Closed-Loop Dissipativity Guarantees](https://mlsanity.com/paper/2404.07373/) (2024)
- [MAD: A Magnitude And Direction Policy Parametrization for Stability Constrained Reinforcement Learning](https://mlsanity.com/paper/2504.02565/) (2025)

## Abstract (arXiv)

We propose a framework for the design of feedback controllers that combines the optimization-driven and model-free advantages of deep reinforcement learning with the stability guarantees provided by using the Youla-Kucera parameterization to define the search domain. Recent advances in behavioral systems allow us to construct a data-driven internal model; this enables an alternative realization of the Youla-Kucera parameterization based entirely on input-output exploration data. Perhaps of independent interest, we formulate and analyze the stability of such data-driven models in the presence of noise. The Youla-Kucera approach requires a stable "parameter" for controller design. For the training of reinforcement learning agents, the set of all stable linear operators is given explicitly through a matrix factorization approach. Moreover, a nonlinear extension is given using a neural network to express a parameterized set of stable operators, which enables seamless integration with standard deep learning libraries. Finally, we show how these ideas can also be applied to tune fixed-structure controllers.