Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
We study reinforcement learning (RL) for a class of continuous-time linear-quadratic (LQ) control problems for diffusions, where states are scalar-valued and running control rewards are absent but volatilities of the state processes depend on both state and control variables. We apply a model-free approach that relies neither on knowledge of model parameters nor on their estimations, and devise an RL algorithm to learn the optimal policy parameter directly. Our main contributions include the introduction of an exploration schedule and a regret analysis of the proposed algorithm. We provide the convergence rate of the policy parameter to the optimal one, and prove that the algorithm achieves a regret bound of O(N^frac34) up to a logarithmic factor, where N is the number of learning episodes. We conduct a simulation study to validate the theoretical results and demonstrate the effectiveness and reliability of the proposed algorithm. We also perform numerical comparisons between our method and those of the recent model-based stochastic LQ RL studies adapted to the state- and control-dependent volatility setting, demonstrating a better performance of the former in terms of regret bounds.
Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 70 references that are in the map. First posted on arXiv 24 Jul 2024.
Builds on
30 of 70 references.
References · 30
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
- 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
- 2019Provably Efficient Exploration in Policy Optimizationcited by 205
- 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
- 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 122
- 2020Naive Exploration is Optimal for Online LQRcited by 116
- 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
- 2022q-Learning in Continuous Timecited by 91
- 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 84
- 2019Learning Linear-Quadratic Regulators Efficiently with only √T Regretcited by 70
18 more
- 2018Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systemscited by 67
- 2019Making Deep Q-learning methods robust to time discretizationcited by 58
- 2018Online Linear Quadratic Controlcited by 56
- 2020Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizoncited by 50
- 2020Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizoncited by 35
- 2020Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controlscited by 31
- 2023A Theoretical Analysis of Optimistic Proximal Policy Optimization in Linear Markov Decision Processescited by 29
- 2022Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problemscited by 25
- 2021Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methodscited by 23
- 2023Actor-Critic learning for mean-field control in continuous timecited by 22
- 2023Continuous-time q-learning for mean-field control problemscited by 22
- 2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 21
- 2021Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex modelscited by 21
- 2020Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Timecited by 20
- 2022Single Time-scale Actor-critic Method to Solve the Linear Quadratic Regulator with Convergence Guaranteescited by 14
- 2021Online Policy Gradient for Model Free Learning of Linear Quadratic Regulators with √T Regretcited by 12
- 2022Global Convergence of Two-timescale Actor-Critic for Solving Linear Quadratic Regulatorcited by 8
- 2024Regret Bounds for Episodic Risk-Sensitive Linear Quadratic Regulatorcited by 1
Built on by
Citing papers per year: 2024: 2; 2025: 4; 2026: 6.
12 papers.
- 2024Regret of exploratory policy improvement and q-learningcited by 12
- 2026ART for Diffusion Sampling: A Reinforcement Learning Approach to Timestep Schedulecited by 4
- 2025Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximationcited by 3
- 2025Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problemscited by 3
- 2024Multi-Task Dynamic Pricing in Credit Market with Contextual Informationcited by 2
- 2025Continuous-Time Reinforcement Learning for Asset-Liability Managementcited by 2
- 2025Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimationcited by 1
- 2026Amortized Guidance for Image Inpainting with Pretrained Diffusion Modelscited by 1
- 2026From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environmentscited by 0
- 2026ART for Diffusion Sampling: Continuous-Time Control and Actor-Critic Learningcited by 0
- 2026Continuous-Time Reinforcement Learning for N-Player Stochastic Differential Games with Exploratory Policiescited by 0
- 2026Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergencecited by 0
- 2026Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence14 Sep 2026
- 2026Continuous-Time Reinforcement Learning for N-Player Stochastic Differential Games with Exploratory Policies22 Jul 2026
- 2026ART for Diffusion Sampling: Continuous-Time Control and Actor-Critic Learning2 Jul 2026
- 2026From Ticks to Flows: Dynamics of Neural Reinforcement Learning in Continuous Environments2 Jun 2026
- 2026Amortized Guidance for Image Inpainting with Pretrained Diffusion Models13 May 2026
- 2026ART for Diffusion Sampling: A Reinforcement Learning Approach to Timestep Schedule26 Jan 2026
- 2025Continuous-Time Reinforcement Learning for Asset-Liability Management27 Sep 2025
- 2025Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimation4 Aug 2025
- 2025Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems1 Jul 2025
- 2025Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation20 May 2025
- 2024Regret of exploratory policy improvement and q-learning2 Nov 2024
- 2024Multi-Task Dynamic Pricing in Credit Market with Contextual Information18 Oct 2024
Similar, not cited
Close in wording, no citation either way
- 2026Posterior Sampling Reinforcement Learning with Gaussian Processes for Continuous Control: Sublinear Regret Bounds for Unbounded State Spaces
- 2024Achieving Constant Regret in Linear Markov Decision Processes
- 2024Efficient, Low-Regret, Online Reinforcement Learning for Linear MDPs
- 2021Fast Rates for the Regret of Offline Reinforcement Learning
- 2024Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Bounds
- 2024Horizon-Free Regret for Linear Markov Decision Processes
- 2025Sample-efficient and Scalable Exploration in Continuous-Time RL
- 2024Local Linearity: the Key for No-regret Reinforcement Learning in Continuous MDPs