mlsanityData to 7 Oct 2026

NeoRL: Efficient Exploration for Nonepisodic RL

arXiv 2406.01175 ↗First posted 3 Jun 2024cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We study the problem of nonepisodic reinforcement learning (RL) for nonlinear dynamical systems, where the system dynamics are unknown and the RL agent has to learn from a single trajectory, i.e., without resets. We propose Nonepisodic Optimistic RL (NeoRL), an approach based on the principle of optimism in the face of uncertainty. NeoRL uses well-calibrated probabilistic models and plans optimistically w.r.t. the epistemic uncertainty about the unknown dynamics. Under continuity and bounded energy assumptions on the system, we provide a first-of-its-kind regret bound of O(GammaT √T) for general nonlinear systems with Gaussian process dynamics. We compare NeoRL to other baselines on several deep RL environments and empirically demonstrate that NeoRL achieves the optimal average cost while incurring the least regret.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 44 of its 73 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

44 of 73 references.

References · 44

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  2. 2016OpenAI Gymcited by 1,684
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2018DeepMind Control Suitecited by 935
  6. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  9. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  10. 2017On Kernelized Multi-armed Banditscited by 348
  11. 2019Contrastive Learning of Structured World Modelscited by 206
  12. 2020On Information Gain and Regret Bounds in Gaussian Process Banditscited by 156
32 more
  1. 2017On the Sample Complexity of the Linear Quadratic Regulatorcited by 151
  2. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 116
  3. 2020Naive Exploration is Optimal for Online LQRcited by 111
  4. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  5. 2020Sample-efficient Cross-Entropy Method for Real-time Planningcited by 90
  6. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 87
  7. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
  8. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
  9. 2020Active Learning for Nonlinear System Identification with Guaranteescited by 78
  10. 2017Lower Bounds on Regret for Noisy Gaussian Process Bandit Optimizationcited by 77
  11. 2019Learning Linear-Quadratic Regulators Efficiently with only √T Regretcited by 67
  12. 2017Learning Unknown Markov Decision Processes: A Thompson Sampling Approachcited by 65
  13. 2020Logarithmic Regret Bound in Partially Observable Linear Dynamical Systemscited by 46
  14. 2020Learning nonlinear dynamical systems from a single trajectorycited by 46
  15. 2020Non-asymptotic and Accurate Learning of Nonlinear Dynamical Systemscited by 39
  16. 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
  17. 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 32
  18. 2023Optimistic Active Exploration of Dynamical Systemscited by 32
  19. 2020Continual Learning of Control Primitives: Skill Discovery via Reset-Gamescited by 29
  20. 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 29
  21. 2019AMZ Driverless: The Full Autonomous Racing Systemcited by 28
  22. 2022Model-based Causal Bayesian Optimizationcited by 26
  23. 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 21
  24. 2023Efficient Exploration in Continuous-time Model-based Reinforcement Learningcited by 21
  25. 2020Average-reward model-free reinforcement learning: a systematic review and literature mappingcited by 19
  26. 2020Efficient Optimistic Exploration in Linear-Quadratic Regulators via Lagrangian Relaxationcited by 18
  27. 2021Average-Reward Reinforcement Learning with Trust Region Methodscited by 18
  28. 2023Optimal Exploration for Model-Based RL in Nonlinear Systemscited by 17
  29. 2017Optimism-Based Adaptive Regulation of Linear-Quadratic Systemscited by 14
  30. 2023Hallucinated Adversarial Control for Conservative Offline Policy Evaluationcited by 13
  31. 2023Off-Policy Average Reward Actor-Critic with Deterministic Policy Searchcited by 10
  32. 2024Data-Enabled Policy Optimization for Direct Adaptive Learning of the LQRcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
  2. 2025No-Regret Thompson Sampling for Finite-Horizon Markov Decision Processes with Gaussian Processescited by 3
  3. 2025Learning Soft Robotic Dynamics with Active Explorationcited by 2