mlsanityData to 14 Oct 2026

Accelerated Online Reinforcement Learning using Auxiliary Start State Distributions

arXiv 2507.04606 ↗First posted 7 Jul 2025cs.LG cs.AI cs.ROcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

A long-standing problem in online reinforcement learning (RL) is of ensuring sample efficiency, which stems from an inability to explore environments efficiently. Most attempts at efficient exploration tackle this problem in a setting where learning begins from scratch, without prior information available to bootstrap learning. However, such approaches fail to leverage expert demonstrations and simulators that can reset to arbitrary states. These affordances are valuable resources that offer enormous potential to guide exploration and speed up learning. In this paper, we explore how a small number of expert demonstrations and a simulator allowing arbitrary resets can accelerate learning during online RL. We find that training with a suitable choice of an auxiliary start state distribution that may differ from the true start state distribution of the underlying Markov Decision Process can significantly improve sample efficiency. We find that using a notion of safety to inform the choice of this auxiliary distribution significantly accelerates learning. By using episode length information as a way to operationalize this notion, we demonstrate state-of-the-art sample efficiency on a sparse-reward hard-exploration environment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 25 references that are in the map. First posted on arXiv 7 Jul 2025.

Builds on

19 of 25 references.

References · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2016Generative Adversarial Imitation Learningcited by 1,597
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  8. 2018Exploration by Random Network Distillationcited by 993
  9. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  11. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  12. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
7 more
  1. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  2. 2020First return, then explorecited by 266
  3. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  4. 2022Jump-Start Reinforcement Learningcited by 112
  5. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  6. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  7. 2021An Experimental Design Perspective on Model-Based Reinforcement Learningcited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning
  2. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  3. 2024Accelerating Inverse Reinforcement Learning with Expert Bootstrapping
  4. 2023Bridging Imitation and Online Reinforcement Learning: An Optimistic Tale
  5. 2025Fill in the Blanks: Accelerating Q-Learning with a Handful of Demonstrations in Sparse Reward Settings
  6. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning
  7. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  8. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data