# MR-ARL: Model Reference Adaptive Reinforcement Learning for Robustly Stable On-Policy Data-Driven LQR

arXiv: https://arxiv.org/abs/2402.14483 · HTML: https://mlsanity.com/paper/2402.14483/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 58 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 14 of 14)

- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018)
- [Formulas for Data-driven Control: Stabilization, Optimality and Robustness](https://mlsanity.com/paper/1903.06842/) (2019)
- [Safely Learning to Control the Constrained Linear Quadratic Regulator](https://mlsanity.com/paper/1809.10121/) (2018)
- [Finite-time Analysis of Approximate Policy Iteration for the Linear Quadratic Regulator](https://mlsanity.com/paper/1905.12842/) (2019)
- [Robust Policy Iteration for Continuous-time Linear Quadratic Regulation](https://mlsanity.com/paper/2005.09528/) (2020)
- [A Historical Perspective of Adaptive Control and Learning](https://mlsanity.com/paper/2108.11336/) (2021)
- [How are policy gradient methods affected by the limits of control?](https://mlsanity.com/paper/2206.06863/) (2022)
- [Efficient Off-Policy Q-Learning for Data-Based Discrete-Time LQR Problems](https://mlsanity.com/paper/2105.07761/) (2021)
- [Low-complexity Learning of Linear Quadratic Regulators from Noisy Data](https://mlsanity.com/paper/2005.01082/) (2020)
- [Online Policies for Real-Time Control Using MRAC-RL](https://mlsanity.com/paper/2103.16551/) (2021)
- [Data-Driven LQR Control Design](https://mlsanity.com/paper/1807.03263/) (2018)
- [Data-driven Linear Quadratic Regulation via Semidefinite Programming](https://mlsanity.com/paper/1911.07767/) (2019)
- [Online Adaptive Optimal Control Algorithm Based on Synchronous Integral Reinforcement Learning With Explorations](https://mlsanity.com/paper/2105.09006/) (2021)

## Abstract (arXiv)

This article introduces a novel framework for data-driven linear quadratic regulator (LQR) design. First, we introduce a reinforcement learning paradigm for on-policy data-driven LQR, where exploration and exploitation are simultaneously performed while guaranteeing robust stability of the whole closed-loop system encompassing the plant and the control/learning dynamics. Then, we propose Model Reference Adaptive Reinforcement Learning (MR-ARL), a control architecture integrating tools from reinforcement learning and model reference adaptive control. The approach stands on a variable reference model containing the currently identified value function. Then, an adaptive stabilizer is used to ensure convergence of the applied policy to the optimal one, convergence of the plant to the optimal reference model, and overall robust closed-loop stability. The proposed framework provides theoretical robustness certificates against real-world perturbations such as measurement noise, plant nonlinearities, or slowly varying parameters. The effectiveness of the proposed architecture is validated via realistic numerical simulations.