# Augmented RBMLE-UCB Approach for Adaptive Control of Linear Quadratic Systems

arXiv: https://arxiv.org/abs/2201.10542 · HTML: https://mlsanity.com/paper/2201.10542/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 68 references that are in the map.
- First posted on arXiv 25 Jan 2022.

## Builds on (most cited first, 15 of 15)

- [On the Sample Complexity of the Linear Quadratic Regulator](https://mlsanity.com/paper/1710.01688/) (2017)
- [Naive Exploration is Optimal for Online LQR](https://mlsanity.com/paper/2001.09576/) (2020)
- [Regret Bounds for Robust Adaptive Control of the Linear Quadratic Regulator](https://mlsanity.com/paper/1805.09388/) (2018)
- [Learning Unknown Markov Decision Processes: A Thompson Sampling Approach](https://mlsanity.com/paper/1709.04570/) (2017)
- [Least-Squares Temporal Difference Learning for the Linear Quadratic Regulator](https://mlsanity.com/paper/1712.08642/) (2017)
- [On Adaptive Linear-Quadratic Regulators](https://mlsanity.com/paper/1806.10749/) (2018)
- [Reinforcement Learning with Fast Stabilization in Linear Dynamical Systems](https://mlsanity.com/paper/2007.12291/) (2020)
- [Input Perturbations for Adaptive Control and Learning](https://mlsanity.com/paper/1811.04258/) (2018)
- [Online Regret Bounds for Undiscounted Continuous Reinforcement Learning](https://mlsanity.com/paper/1302.2550/) (2013)
- [Thompson Sampling for Linear-Quadratic Control Problems](https://mlsanity.com/paper/1703.08972/) (2017)
- [Model-Free Linear Quadratic Control via Reduction to Expert Prediction](https://mlsanity.com/paper/1804.06021/) (2018)
- [Efficient Reinforcement Learning for High Dimensional Linear Quadratic Systems](https://mlsanity.com/paper/1303.5984/) (2013)
- [Reward Biased Maximum Likelihood Estimation for Reinforcement Learning](https://mlsanity.com/paper/2011.07738/) (2020)
- [Reward-Biased Maximum Likelihood Estimation for Linear Stochastic Bandits](https://mlsanity.com/paper/2010.04091/) (2020)
- [Infinite-horizon Risk-constrained Linear Quadratic Regulator with Average Cost](https://mlsanity.com/paper/2103.15363/) (2021)

## Built on by (most cited first, 5 of 5)

- [Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration](https://mlsanity.com/paper/2305.18258/) (2023)
- [Learning the Uncertainty Sets for Control Dynamics via Set Membership: A Non-Asymptotic Analysis](https://mlsanity.com/paper/2309.14648/) (2023)
- [Value-Biased Maximum Likelihood Estimation for Model-based Reinforcement Learning in Discounted Linear MDPs](https://mlsanity.com/paper/2310.11515/) (2023)
- [The Confusing Instance Principle for Online Linear Quadratic Control](https://mlsanity.com/paper/2510.19531/) (2025)
- [Finite Time Regret Bounds for Minimum Variance Control of Autoregressive Systems with Exogenous Inputs](https://mlsanity.com/paper/2305.16974/) (2023)

## Abstract (arXiv)

We consider the problem of controlling an unknown stochastic linear system with quadratic costs - called the adaptive LQ control problem. We re-examine an approach called ''Reward Biased Maximum Likelihood Estimate'' (RBMLE) that was proposed more than forty years ago, and which predates the ''Upper Confidence Bound'' (UCB) method as well as the definition of ''regret'' for bandit problems. It simply added a term favoring parameters with larger rewards to the criterion for parameter estimation. We show how the RBMLE and UCB methods can be reconciled, and thereby propose an Augmented RBMLE-UCB algorithm that combines the penalty of the RBMLE method with the constraints of the UCB method, uniting the two approaches to optimism in the face of uncertainty. We establish that theoretically, this method retains TildeO(√T) regret, the best-known so far. We further compare the empirical performance of the proposed Augmented RBMLE-UCB and the standard RBMLE (without the augmentation) with UCB, Thompson Sampling, Input Perturbation, Randomized Certainty Equivalence and StabL on many real-world examples including flight control of Boeing 747 and Unmanned Aerial Vehicle. We perform extensive simulation studies showing that the Augmented RBMLE consistently outperforms UCB, Thompson Sampling and StabL by a huge margin, while it is marginally better than Input Perturbation and moderately better than Randomized Certainty Equivalence.