mlsanityData to 7 Oct 2026

Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments

arXiv 2304.09825 ↗First posted 18 Apr 2023cs.LG cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

One of the key challenges of Reinforcement Learning (RL) is the ability of agents to generalise their learned policy to unseen settings. Moreover, training RL agents requires large numbers of interactions with the environment. Motivated by the recent success of Offline RL and Imitation Learning (IL), we conduct a study to investigate whether agents can leverage offline data in the form of trajectories to improve the sample-efficiency in procedurally generated environments. We consider two settings of using IL from offline data for RL: (1) pre-training a policy before online RL training and (2) concurrently training a policy with online RL and IL from offline data. We analyse the impact of the quality (optimality of trajectories) and diversity (number of trajectories and covered level) of available offline trajectories on the effectiveness of both approaches. Across four well-known sparse reward tasks in the MiniGrid environment, we find that using IL for pre-training and concurrently during online RL training both consistently improve the sample-efficiency while converging to optimal policies. Furthermore, we show that pre-training a policy from as few as two trajectories can make the difference between learning an optimal policy at the end of online training and not learning at all. Our findings motivate the widespread adoption of IL for pre-training and concurrent IL in procedurally generated environments whenever offline trajectories are available or can be generated.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 64 references that are in the map. First posted on arXiv 18 Apr 2023.

Builds on

43 of 64 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  2. 2022Boosting Offline Reinforcement Learning via Data Rebalancing17 Oct
  3. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?12 Apr
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun

Other references · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  5. 2022A Generalist Agentcited by 802
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  9. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  10. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  11. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  12. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
26 more
  1. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  2. 2017Count-Based Exploration with Neural Density Modelscited by 255
  3. 2020Prioritized Level Replaycited by 190
  4. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  5. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  6. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  7. 2018Self-Imitation Learningcited by 135
  8. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  9. 2020Phasic Policy Gradientcited by 121
  10. 2021APS: Active Pretraining with Successor Featurescited by 109
  11. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  12. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 79
  13. 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 77
  14. 2021What Matters for Adversarial Imitation Learning?cited by 70
  15. 2021Adversarially Guided Actor-Criticcited by 47
  16. 2021AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scalecited by 43
  17. 2021Same State, Different Task: Continual Reinforcement Learning without Interferencecited by 42
  18. 2020BeBold: Exploration Beyond the Boundary of Explored Regionscited by 32
  19. 2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
  20. 2021Decoupled Reinforcement Learning to Stabilise Intrinsically-Motivated Explorationcited by 25
  21. 2021Measuring Sample Efficiency and Generalization in Reinforcement Learning Benchmarks: NeurIPS 2020 Procgen Benchmarkcited by 23
  22. 2022Improving Policy Optimization with Generalist-Specialist Learningcited by 19
  23. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  24. 2022Leveraging Fully Observable Policies for Learning under Partial Observabilitycited by 17
  25. 2019I'm sorry Dave, I'm afraid I can't do that, Deep Q-learning from forbidden actioncited by 4
  26. 2022Towards Improving Exploration in Self-Imitation Learning using Intrinsic Motivationcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Highway Graph to Accelerate Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Offline Reinforcement Learning Hands-On
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  3. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  4. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  5. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
  6. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  7. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  8. 2023The Generalization Gap in Offline Reinforcement Learning