# Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon

arXiv: https://arxiv.org/abs/2006.15316 · HTML: https://mlsanity.com/paper/2006.15316/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jun 2020.

## Built on by (most cited first, 30 of 35)

- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms](https://mlsanity.com/paper/2111.11232/) (2021)
- [Policy Optimization for Continuous Reinforcement Learning](https://mlsanity.com/paper/2305.18901/) (2023)
- [Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games](https://mlsanity.com/paper/2107.13090/) (2021)
- [Reinforcement learning for linear-convex models with jumps via stability analysis of feedback controls](https://mlsanity.com/paper/2104.09311/) (2021)
- [Efficient Exploration in Continuous-time Model-based Reinforcement Learning](https://mlsanity.com/paper/2310.19848/) (2023)
- [Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models](https://mlsanity.com/paper/2112.10264/) (2021)
- [Fast Policy Learning for Linear Quadratic Control with Entropy Regularization](https://mlsanity.com/paper/2311.14168/) (2023)
- [Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning](https://mlsanity.com/paper/2503.09981/) (2025)
- [Entropy annealing for policy mirror descent in continuous time and space](https://mlsanity.com/paper/2405.20250/) (2024)
- [Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems](https://mlsanity.com/paper/2407.17226/) (2024)
- [Exploratory Optimal Stopping: A Singular Control Formulation](https://mlsanity.com/paper/2408.09335/) (2024)
- [Reinforcement Learning for Adaptive Optimal Stationary Control of Linear Stochastic Systems](https://mlsanity.com/paper/2107.07788/) (2021)
- [Logarithmic regret bounds for continuous-time average-reward Markov decision processes](https://mlsanity.com/paper/2205.11168/) (2022)
- [Optimal scheduling of entropy regulariser for continuous-time linear-quadratic reinforcement learning](https://mlsanity.com/paper/2208.04466/) (2022)
- [PhiBE: A PDE-based Bellman Equation for Continuous Time Policy Evaluation](https://mlsanity.com/paper/2405.12535/) (2024)
- [Square-root regret bounds for continuous-time episodic Markov decision processes](https://mlsanity.com/paper/2210.00832/) (2022)
- [Statistical Learning with Sublinear Regret of Propagator Models](https://mlsanity.com/paper/2301.05157/) (2023)
- [Data-driven rules for multidimensional reflection problems](https://mlsanity.com/paper/2311.06639/) (2023)
- [Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Control](https://mlsanity.com/paper/2504.21793/) (2025)
- [Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation](https://mlsanity.com/paper/2505.14821/) (2025)
- [Bayesian Algorithms Learn to Stabilize Unknown Continuous-Time Systems](https://mlsanity.com/paper/2112.15094/) (2021)
- [Continuous-Time Reinforcement Learning for Asset-Liability Management](https://mlsanity.com/paper/2509.23280/) (2025)
- [Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-off](https://mlsanity.com/paper/2212.08949/) (2022)
- [ε-Policy Gradient for Online Pricing](https://mlsanity.com/paper/2405.03624/) (2024)
- [Interacting Particle Systems for Fast Linear Quadratic RL](https://mlsanity.com/paper/2406.11057/) (2024)
- [Learning to steer with Brownian noise](https://mlsanity.com/paper/2410.03221/) (2024)
- [Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimation](https://mlsanity.com/paper/2508.02103/) (2025)
- [Reinforcement Learning Policies in Continuous-Time Linear Systems](https://mlsanity.com/paper/2109.07630/) (2021)
- [Regret Analysis of Certainty Equivalence Policies in Continuous-Time Linear-Quadratic Systems](https://mlsanity.com/paper/2206.04434/) (2022)

## Abstract (arXiv)

We study finite-time horizon continuous-time linear-quadratic reinforcement learning problems in an episodic setting, where both the state and control coefficients are unknown to the controller. We first propose a least-squares algorithm based on continuous-time observations and controls, and establish a logarithmic regret bound of order O((ln M)(lnln M)), with M being the number of learning episodes. The analysis consists of two parts: perturbation analysis, which exploits the regularity and robustness of the associated Riccati differential equation; and parameter estimation error, which relies on sub-exponential properties of continuous-time least-squares estimators. We further propose a practically implementable least-squares algorithm based on discrete-time observations and piecewise constant controls, which achieves similar logarithmic regret with an additional term depending explicitly on the time stepsizes used in the algorithm.