# Scalable regret for learning to control network-coupled subsystems with unknown dynamics

arXiv: https://arxiv.org/abs/2108.07970 · HTML: https://mlsanity.com/paper/2108.07970/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 18 Aug 2021.

## Builds on (most cited first, 26 of 26)

- [Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents](https://mlsanity.com/paper/1802.08757/) (2018)
- [Naive Exploration is Optimal for Online LQR](https://mlsanity.com/paper/2001.09576/) (2020)
- [Regret Bounds for Robust Adaptive Control of the Linear Quadratic Regulator](https://mlsanity.com/paper/1805.09388/) (2018)
- [Learning Linear-Quadratic Regulators Efficiently with only √T Regret](https://mlsanity.com/paper/1902.06223/) (2019)
- [Near-optimal Reinforcement Learning in Factored MDPs](https://mlsanity.com/paper/1403.3741/) (2014)
- [Distributed Reinforcement Learning for Decentralized Linear Quadratic Control: A Derivative-Free Policy Optimization Approach](https://mlsanity.com/paper/1912.09135/) (2019)
- [Multi Type Mean Field Reinforcement Learning](https://mlsanity.com/paper/2002.02513/) (2020)
- [Decentralized Multi-Agent Reinforcement Learning with Networked Agents: Recent Advances](https://mlsanity.com/paper/1912.03821/) (2019)
- [On Adaptive Linear-Quadratic Regulators](https://mlsanity.com/paper/1806.10749/) (2018)
- [Input Perturbations for Adaptive Control and Learning](https://mlsanity.com/paper/1811.04258/) (2018)
- [Efficient Optimistic Exploration in Linear-Quadratic Regulators via Lagrangian Relaxation](https://mlsanity.com/paper/2007.06482/) (2020)
- [Efficient Reinforcement Learning in Factored MDPs with Application to Constrained RL](https://mlsanity.com/paper/2008.13319/) (2020)
- [Optimism-Based Adaptive Regulation of Linear-Quadratic Systems](https://mlsanity.com/paper/1711.07230/) (2017)
- [Certainty Equivalence is Efficient for Linear Quadratic Control](https://mlsanity.com/paper/1902.07826/) (2019)
- [Policy Gradient-based Algorithms for Continuous-time Linear Quadratic Control](https://mlsanity.com/paper/2006.09178/) (2020)
- [Bayesian Optimal Control of Smoothly Parameterized Systems: The Lazy Posterior Sampling Algorithm](https://mlsanity.com/paper/1406.3926/) (2014)
- [Learning-based Control of Unknown Linear Systems with Thompson Sampling](https://mlsanity.com/paper/1709.04047/) (2017)
- [Reinforcement Learning in Non-Stationary Discrete-Time Linear-Quadratic Mean-Field Games](https://mlsanity.com/paper/2009.04350/) (2020)
- [Finite Time Adaptive Stabilization of LQ Systems](https://mlsanity.com/paper/1807.09120/) (2018)
- [Learning Distributed Stabilizing Controllers for Multi-Agent Systems](https://mlsanity.com/paper/2103.04480/) (2021)
- [Distributed Q-Learning for Dynamically Decoupled Systems](https://mlsanity.com/paper/1809.08745/) (2018)
- [Thompson sampling for linear quadratic mean-field teams](https://mlsanity.com/paper/2011.04686/) (2020)
- [Team-Optimal Solution of Finite Number of Mean-Field Coupled LQG Subsystems](https://mlsanity.com/paper/2012.02052/) (2020)
- [Distributed Q-Learning with State Tracking for Multi-agent Networked Control](https://mlsanity.com/paper/2012.12383/) (2020)
- [Linear Quadratic Mean Field Teams: Optimal and Approximately Optimal Decentralized Solutions](https://mlsanity.com/paper/1609.00056/) (2016)
- [Optimal control of network-coupled subsystems: Spectral decomposition and low-dimensional solutions](https://mlsanity.com/paper/2009.12367/) (2020)

## Built on by (most cited first, 2 of 2)

- [Joint Learning of Linear Time-Invariant Dynamical Systems](https://mlsanity.com/paper/2112.10955/) (2021)
- [Joint Learning-Based Stabilization of Multiple Unknown Linear Systems](https://mlsanity.com/paper/2201.01387/) (2022)

## Abstract (arXiv)

We consider the problem of controlling an unknown linear quadratic Gaussian (LQG) system consisting of multiple subsystems connected over a network. Our goal is to minimize and quantify the regret (i.e. loss in performance) of our strategy with respect to an oracle who knows the system model. Viewing the interconnected subsystems globally and directly using existing LQG learning algorithms for the global system results in a regret that increases super-linearly with the number of subsystems. Instead, we propose a new Thompson sampling based learning algorithm which exploits the structure of the underlying network. We show that the expected regret of the proposed algorithm is bounded by tildeO big( n √T big) where n is the number of subsystems, T is the time horizon and the tildeO(·) notation hides logarithmic terms in n and T. Thus, the regret scales linearly with the number of subsystems. We present numerical experiments to illustrate the salient features of the proposed algorithm.