Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems
Not on the map. Why?
Abstract (arXiv)
We study reinforcement learning (RL) for the same class of continuous-time stochastic linear--quadratic (LQ) control problems as in citehuang2024sublinear, where volatilities depend on both states and controls while states are scalar-valued and running control rewards are absent. We propose a model-free, data-driven exploration mechanism that adaptively adjusts entropy regularization by the critic and policy variance by the actor. Unlike the constant or deterministic exploration schedules employed in citehuang2024sublinear, which require extensive tuning for implementations and ignore learning progresses during iterations, our adaptive exploratory approach boosts learning efficiency with minimal tuning. Despite its flexibility, our method achieves a sublinear regret bound that matches the best-known model-free results for this class of LQ problems, which were previously derived only with fixed exploration schedules. Numerical experiments demonstrate that adaptive explorations accelerate convergence and improve regret performance compared to the non-adaptive model-free and model-based counterparts.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 51 references that are in the map. First posted on arXiv 1 Jul 2025.
Builds on
23 of 51 references.
References · 23
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
- 2018Exploration by Random Network Distillationcited by 966
- 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
- 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
- 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
- 2020First return, then explorecited by 256
- 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
- 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
- 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 102
- 2020Fast active learning for pure exploration in reinforcement learningcited by 92
- 2017Surprise-Based Intrinsic Motivation for Deep Reinforcement Learningcited by 83
- 2019A survey on intrinsic motivation in reinforcement learningcited by 79
11 more
- 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 76
- 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 62
- 2019Making Deep Q-learning methods robust to time discretizationcited by 51
- 2021Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methodscited by 21
- 2021Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex modelscited by 18
- 2024Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problemscited by 10
- 2024Regret of exploratory policy improvement and q-learningcited by 10
- 2024Mean--Variance Portfolio Selection by Continuous-Time Reinforcement Learning: Algorithms, Regret Analysis, and Empirical Studycited by 7
- 2019Stabilization Control for ItO Stochastic System with Indefinite State and Control Weight Costscited by 1
- 2021Indefinite linear-quadratic optimal control of mean-field stochastic differential equation with jump diffusion: an equivalent cost functional methodcited by 1
- 2024Stochastic linear quadratic optimal control problems with regime-switching jumps in infinite horizoncited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.