mlsanityData to 14 Oct 2026

Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimation

arXiv 2508.02103 ↗First posted 4 Aug 2025cs.LG stat.MLcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Continuous-time reinforcement learning (CTRL) provides a natural framework for sequential decision-making in dynamic environments where interactions evolve continuously over time. While CTRL has shown growing empirical success, its ability to adapt to varying levels of problem difficulty remains poorly understood. In this work, we investigate the instance-dependent behavior of CTRL and introduce a simple, model-based algorithm built on maximum likelihood estimation (MLE) with a general function approximator. Unlike existing approaches that estimate system dynamics directly, our method estimates the state marginal density to guide learning. We establish instance-dependent performance guarantees by deriving a regret bound that scales with the total reward variance and measurement resolution. Notably, the regret becomes independent of the specific measurement strategy when the observation frequency adapts appropriately to the problem's complexity. To further improve performance, our algorithm incorporates a randomized measurement schedule that enhances sample efficiency without increasing measurement cost. These results highlight a new direction for designing CTRL algorithms that automatically adjust their learning behavior based on the underlying difficulty of the environment.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 41 of its 64 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

41 of 64 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximation20 May
  2. 2024Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Bounds16 Aug
  3. 2024Sublinear Regret for a Class of Continuous-Time Linear-Quadratic Reinforcement Learning Problems24 Jul
  4. 2024When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL3 Jun
  5. 2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning11 Feb

Other references · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Neural Ordinary Differential Equationscited by 3,779
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2016OpenAI Gymcited by 1,736
  5. 2019Hamiltonian Neural Networkscited by 712
  6. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  7. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 384
  8. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  9. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 223
  10. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 200
  11. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 195
  12. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
24 more
  1. 2019Non-Asymptotic Gap-Dependent Regret Bounds for Tabular MDPscited by 105
  2. 2020Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mappingcited by 102
  3. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  4. 2023DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuningcited by 84
  5. 2020Logarithmic Regret for Reinforcement Learning with Linear Function Approximationcited by 79
  6. 2021Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discriminationcited by 53
  7. 2022Computationally Efficient Horizon-Free Reinforcement Learning for Linear Mixture MDPscited by 52
  8. 2021Fine-Grained Gap-Dependent Bounds for Tabular MDPs via Adaptive Multi-Step Bootstrapcited by 49
  9. 2021Improved Variance-Aware Confidence Sets for Linear Bandits and Linear Mixture MDPcited by 45
  10. 2021Improved Regret Analysis for Variance-Adaptive Linear Bandits and Horizon-Free Linear Mixture MDPscited by 44
  11. 2021Continuous-Time Model-Based Reinforcement Learningcited by 41
  12. 2023Variance-Dependent Regret Bounds for Linear Bandits and Reinforcement Learning: Adaptivity and Computational Efficiencycited by 40
  13. 2020Unsupervised Learning of Lagrangian Dynamics from Images for Prediction and Controlcited by 38
  14. 2021First-Order Regret in Reinforcement Learning with Linear Function Approximation: A Robust Estimation Approachcited by 38
  15. 2020Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizoncited by 35
  16. 2021Beyond Value-Function Gaps: Improved Instance-Dependent Regret Bounds for Episodic Reinforcement Learningcited by 34
  17. 2022Horizon-Free Reinforcement Learning in Polynomial Time: the Power of Stationary Policiescited by 27
  18. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  19. 2023Efficient Exploration in Continuous-time Model-based Reinforcement Learningcited by 22
  20. 2023Sharp Variance-Dependent Bounds in Reinforcement Learning: Best of Both Worlds in Stochastic and Deterministic Environmentscited by 19
  21. 2021Value Iteration in Continuous Actions, States and Timecited by 18
  22. 2023Provable benefits of score matchingcited by 18
  23. 2021Uniform-PAC Bounds for Reinforcement Learning with Linear Function Approximationcited by 14
  24. 2023Censored Sampling of Diffusion Models Using 3 Minutes of Human Feedbackcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Breaking the Total Variance Barrier: Sharp Sample Complexity for Linear Heteroscedastic Bandits with Fixed Action Setcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Maximum Likelihood Reinforcement Learning
  2. 2025Sample-efficient and Scalable Exploration in Continuous-Time RL
  3. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  4. 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees
  5. 2026Model-Based Reinforcement Learning for Control under Time-Varying Dynamics
  6. 2024Model-Free Active Exploration in Reinforcement Learning
  7. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confounding
  8. 2026Provably Efficient Reinforcement Learning in Continuous-Time Episodic MDPs with Poisson Decision Epochs