# Reinforcement Learning Policies in Continuous-Time Linear Systems

arXiv: https://arxiv.org/abs/2109.07630 · HTML: https://mlsanity.com/paper/2109.07630/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 45 references that are in the map.
- First posted on arXiv 16 Sep 2021.

## Builds on (most cited first, 13 of 13)

- [On the Sample Complexity of the Linear Quadratic Regulator](https://mlsanity.com/paper/1710.01688/) (2017)
- [Logarithmic Regret Bound in Partially Observable Linear Dynamical Systems](https://mlsanity.com/paper/2003.11227/) (2020)
- [Black-Box Control for Linear Dynamical Systems](https://mlsanity.com/paper/2007.06650/) (2020)
- [Logarithmic Regret for Learning Linear Quadratic Regulators Efficiently](https://mlsanity.com/paper/2002.08095/) (2020)
- [Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon](https://mlsanity.com/paper/2006.15316/) (2020)
- [On Adaptive Linear-Quadratic Regulators](https://mlsanity.com/paper/1806.10749/) (2018)
- [Input Perturbations for Adaptive Control and Learning](https://mlsanity.com/paper/1811.04258/) (2018)
- [Optimism-Based Adaptive Regulation of Linear-Quadratic Systems](https://mlsanity.com/paper/1711.07230/) (2017)
- [On Applications of Bootstrap in Continuous Space Reinforcement Learning](https://mlsanity.com/paper/1903.05803/) (2019)
- [Finite Time Adaptive Stabilization of LQ Systems](https://mlsanity.com/paper/1807.09120/) (2018)
- [Randomized Algorithms for Data-Driven Stabilization of Stochastic Linear Systems](https://mlsanity.com/paper/1905.06978/) (2019)
- [Regret Bounds for Decentralized Learning in Cooperative Multi-Agent Dynamical Systems](https://mlsanity.com/paper/2001.10122/) (2020)
- [A relaxed technical assumption for posterior sampling-based reinforcement learning for control of unknown linear systems](https://mlsanity.com/paper/2108.08502/) (2021)

## Abstract (arXiv)

Linear dynamical systems that obey stochastic differential equations are canonical models. While optimal control of known systems has a rich literature, the problem is technically hard under model uncertainty and there are hardly any results. We initiate study of this problem and aim to learn (and simultaneously deploy) optimal actions for minimizing a quadratic cost function. Indeed, this work is the first that comprehensively addresses the crucial challenge of balancing exploration versus exploitation in continuous-time systems. We present online policies that learn optimal actions fast by carefully randomizing the parameter estimates, and establish their performance guarantees: a regret bound that grows with square-root of time multiplied by the number of parameters. Implementation of the policy for a flight-control task demonstrates its efficacy. Further, we prove sharp stability results for inexact system dynamics and tightly specify the infinitesimal regret caused by sub-optimal actions. To obtain the results, we conduct a novel eigenvalue-sensitivity analysis for matrix perturbation, establish upper-bounds for comparative ratios of stochastic integrals, and introduce the new method of policy differentiation. Our analysis sheds light on fundamental challenges in continuous-time reinforcement learning and suggests a useful cornerstone for similar problems.