mlsanityData to 7 Oct 2026

The Challenges of Exploration for Offline Reinforcement Learning

arXiv 2201.11861 ↗First posted 27 Jan 2022cs.LGcited by 26

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement Learning (ORL) enablesus to separately study the two interlinked processes of reinforcement learning: collecting informative experience and inferring optimal behaviour. The second step has been widely studied in the offline setting, but just as critical to data-efficient RL is the collection of informative data. The task-agnostic setting for data collection, where the task is not known a priori, is of particular interest due to the possibility of collecting a single dataset and using it to solve several downstream tasks as they arise. We investigate this setting via curiosity-based intrinsic motivation, a family of exploration methods which encourage the agent to explore those states or transitions it has not yet learned to model. With Explore2Offline, we propose to evaluate the quality of collected data by transferring the collected data and inferring policies with reward relabelling and standard offline RL algorithms. We evaluate a wide variety of data collection strategies, including a new exploration agent, Intrinsic Model Predictive Control (IMPC), using this scheme and demonstrate their performance on various tasks. We use this decoupled framework to strengthen intuitions about exploration and the data prerequisites for effective offline RL.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 40 of its 50 references that are in the map. First posted on arXiv 27 Jan 2022.

Builds on

40 of 50 references.

Closest ancestors · 6

Same direction, newest first

  1. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  2. 2020Critic Regularized Regression26 Jun
  3. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 34

  1. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  2. 2017Hindsight Experience Replaycited by 1,096
  3. 2018Exploration by Random Network Distillationcited by 966
  4. 2018DeepMind Control Suitecited by 935
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  7. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  8. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  9. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  10. 2018Maximum a Posteriori Policy Optimisationcited by 318
  11. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  12. 2019Self-Supervised Exploration via Disagreementcited by 263
22 more
  1. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  2. 2021Reinforcement Learning with Prototypical Representationscited by 179
  3. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  4. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  5. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  6. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 131
  7. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  8. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  9. 2020Adaptive Reward-Free Explorationcited by 77
  10. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
  11. 2020Generalized Hindsight for Reinforcement Learningcited by 51
  12. 2019Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learningcited by 40
  13. 2021Interesting Object, Curious Agent: Learning Task-Agnostic Explorationcited by 38
  14. 2019Scaling data-driven robotics with reward sketching and batch reinforcement learningcited by 18
  15. 2021Collect & Infer -- a fresh look at data-efficient Reinforcement Learningcited by 17
  16. 2021Evaluating model-based planning and planner amortization for continuous controlcited by 15
  17. 2021Decoupled Exploration and Exploitation Policies for Sample-Efficient Reinforcement Learningcited by 14
  18. 2021Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Explorationcited by 14
  19. 2020Representation Matters: Improving Perception and Exploration for Roboticscited by 11
  20. 2021DisCo RL: Distribution-Conditioned Reinforcement Learning for General-Purpose Policiescited by 9
  21. 2021Efficient Self-Supervised Data Collection for Offline Robot Learningcited by 8
  22. 2019Evaluating task-agnostic exploration for fixed-batch learning of arbitrary future taskscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 10; 2024: 5; 2025: 3; 2026: 1.

26 papers.

Sort:

By area:

  1. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  2. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  3. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  4. 2022Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulationcited by 29
  5. 2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
  6. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  7. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  8. 2024Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learningcited by 14
  9. 2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12
  10. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothnesscited by 12
  11. 2022Exploring through Random Curiosity with General Value Functionscited by 10
  12. 2023Replay across Experiments: A Natural Extension of Off-Policy RLcited by 10
14 more
  1. 2023Think Before You Act: Unified Policy for Interleaving Language Reasoning with Actionscited by 8
  2. 2022Epistemic Monte Carlo Tree Searchcited by 7
  3. 2022Reliable Conditioning of Behavioral Cloning for Offline Reinforcement Learningcited by 6
  4. 2023Offline Reinforcement Learning with Imbalanced Datasetscited by 6
  5. 2023Investigating the role of model-based learning in exploration and transfercited by 5
  6. 2023Goal-conditioned Offline Planning from Curious Explorationcited by 3
  7. 2025Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsitycited by 2
  8. 2024Real-World Fluid Directed Rigid Body Control via Deep Reinforcement Learningcited by 1
  9. 2024Iterative Batch Reinforcement Learning via Safe Diversified Model-based Policy Searchcited by 1
  10. 2025Exploiting Policy Idling for Dexterous Manipulationcited by 1
  11. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Datacited by 0
  12. 2024Online Self-Preferring Language Modelscited by 0
  13. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Modelcited by 0
  14. 2026Efficient Anti-exploration via VQVAE and Fuzzy Clustering in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Offline Reinforcement Learning as Anti-Exploration
  2. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
  3. 2023CUDC: A Curiosity-Driven Unsupervised Data Collection Method with Adaptive Temporal Distances for Offline Reinforcement Learning
  4. 2024A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learning
  5. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  6. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  7. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  8. 2020Offline Reinforcement Learning Hands-On