mlsanityData to 14 Oct 2026

Efficient Exploration in Continuous-time Model-based Reinforcement Learning

arXiv 2310.19848 ↗First posted 30 Oct 2023cs.LG cs.RO math.OCcited by 22

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning algorithms typically consider discrete-time dynamics, even though the underlying systems are often continuous in time. In this paper, we introduce a model-based reinforcement learning algorithm that represents continuous-time dynamics using nonlinear ordinary differential equations (ODEs). We capture epistemic uncertainty using well-calibrated probabilistic models, and use the optimistic principle for exploration. Our regret bounds surface the importance of the measurement selection strategy(MSS), since in continuous time we not only must decide how to explore, but also when to observe the underlying system. Our analysis demonstrates that the regret is sublinear when modeling ODEs with Gaussian Processes (GP) for common choices of MSS, such as equidistant sampling. Additionally, we propose an adaptive, data-dependent, practical MSS that, when combined with GP dynamics, also achieves sublinear regret with significantly fewer samples. We showcase the benefits of continuous-time modeling over its discrete-time counterpart, as well as our proposed adaptive MSS over standard baselines, on several applications.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 36 of its 83 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

36 of 83 references.

References · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Neural Ordinary Differential Equationscited by 3,779
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2016OpenAI Gymcited by 1,736
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  7. 2019Hamiltonian Neural Networkscited by 712
  8. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 422
  9. 2020Lagrangian Neural Networkscited by 398
  10. 2017On Kernelized Multi-armed Banditscited by 352
  11. 2013Learning to Optimize Via Posterior Samplingcited by 256
  12. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
24 more
  1. 2014An Information-Theoretic Analysis of Thompson Samplingcited by 178
  2. 2020On Information Gain and Regret Bounds in Gaussian Process Banditscited by 158
  3. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 140
  4. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  5. 2020Naive Exploration is Optimal for Online LQRcited by 116
  6. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 90
  7. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 86
  8. 2018Information Directed Sampling and Bandits with Heteroscedastic Noisecited by 76
  9. 2019Making Deep Q-learning methods robust to time discretizationcited by 58
  10. 2022A Framework and Benchmark for Deep Batch Active Learning for Regressioncited by 49
  11. 2019Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problemcited by 46
  12. 2021Continuous-Time Model-Based Reinforcement Learningcited by 41
  13. 2021Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learningcited by 38
  14. 2019On Online Learning in Kernelized Markov Decision Processescited by 35
  15. 2020Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizoncited by 35
  16. 2020Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEscited by 32
  17. 2019An Online Learning Approach to Model Predictive Controlcited by 30
  18. 2022Model-based Causal Bayesian Optimizationcited by 26
  19. 2019Feedback Linearization based on Gaussian Processes with event-triggered Online Learningcited by 24
  20. 2013Model-Based Bayesian Explorationcited by 21
  21. 2021Value Iteration in Continuous Actions, States and Timecited by 18
  22. 2023Hallucinated Adversarial Control for Conservative Offline Policy Evaluationcited by 13
  23. 2022Myriad: a real-world testbed to bridge trajectory optimization and deep learningcited by 5
  24. 2021Distributional Gradient Matching for Learning Uncertain Neural Dynamics Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 11; 2026: 7.

22 papers.

Sort:

By area:

  1. 2025Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuningcited by 17
  2. 2024When to Sense and Control? A Time-adaptive Approach for Continuous-Time RLcited by 9
  3. 2024Bridging the Sim-to-Real Gap with Bayesian Inferencecited by 8
  4. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and Statecited by 8
  5. 2025SOMBRL: Scalable and Optimistic Model-Based RLcited by 7
  6. 2026Verifier-Constrained Flow Expansion for Discovery Beyond the Datacited by 7
  7. 2024NeoRL: Efficient Exploration for Nonepisodic RLcited by 4
  8. 2024Goal-Conditioned Data Augmentation for Offline Reinforcement Learningcited by 4
  9. 2025Optimal Control of Probabilistic Dynamics Models via Mean Hamiltonian Minimizationcited by 3
  10. 2025Reconciling Discrete-Time Mixed Policies and Continuous-Time Relaxed Controls in Reinforcement Learning and Stochastic Controlcited by 3
  11. 2025Sample and Computationally Efficient Continuous-Time Reinforcement Learning with General Function Approximationcited by 3
  12. 2025Accuracy, Memory Efficiency and Generalization: A Comparative Study on Liquid Neural Networks and Recurrent Neural Networkscited by 3
10 more
  1. 2025Sample-efficient and Scalable Exploration in Continuous-Time RLcited by 3
  2. 2026Constrained Flow Optimization via Sequential Fine Tuning for Molecular Designcited by 3
  3. 2025Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimationcited by 1
  4. 2026Optimistic Online LQR via Intrinsic Rewardscited by 1
  5. 2025RAD: Retrieval High-quality Demonstrations to Enhance Decision-makingcited by 0
  6. 2025Simulation Priors for Data-Efficient Deep Learningcited by 0
  7. 2026Neural Nonmyopic Bayesian Optimization in Dynamic Cost Settingscited by 0
  8. 2026Search Inspired Exploration in Reinforcement Learningcited by 0
  9. 2026Interaction-Limited Safe Continuous-Time RL for Dynamical Medical Treatmentcited by 0
  10. 2026Provably Efficient Reinforcement Learning in Continuous-Time Episodic MDPs with Poisson Decision Epochscited by 0