Data to 7 Oct 2026

Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning

arXiv 2201.13425 ↗First posted 31 Jan 2022cs.LG cs.AIcited by 93

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Recent progress in deep learning has relied on access to large and diverse datasets. Such data-driven progress has been less evident in offline reinforcement learning (RL), because offline RL data is usually collected to optimize specific target tasks limiting the data's diversity. In this work, we propose Exploratory data for Offline RL (ExORL), a data-centric approach to offline RL. ExORL first generates data with unsupervised reward-free exploration, then relabels this data with a downstream reward before training a policy with offline RL. We find that exploratory data allows vanilla off-policy RL algorithms, without any offline-specific modifications, to outperform or match state-of-the-art offline RL algorithms on downstream tasks. Our findings suggest that data generation is as important as algorithmic advances for offline RL and hence requires careful consideration from the community. Code and data can be found at https://github.com/denisyarats/exorl .

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 47 of its 56 references that are in the map. First posted on arXiv 31 Jan 2022.

Builds on

47 of 56 references.

Closest ancestors · 13

Same direction, newest first

2022The Challenges of Exploration for Offline Reinforcement Learning27 Jan
2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning26 Oct
2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning16 Sep
2021Offline RL Without Off-Policy Evaluation16 Jun
2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
2021Regularized Behavior Value Estimation17 Mar
2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
2020Critic Regularized Regression26 Jun
2020On Reward-Free Reinforcement Learning with Linear Function Approximation19 Jun
2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

1 more

2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 34

2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2020Language Models are Few-Shot Learnerscited by 32,199
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
2013Playing Atari with Deep Reinforcement Learningcited by 3,211
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
2017Hindsight Experience Replaycited by 1,096
2018Exploration by Random Network Distillationcited by 966
2018DeepMind Control Suitecited by 935

22 more

2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
2020Planning to Explore via Self-Supervised World Modelscited by 365
2019Self-Supervised Exploration via Disagreementcited by 263
2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
2019Efficient Exploration via State Marginal Matchingcited by 186
2021Reinforcement Learning with Prototypical Representationscited by 179
2020Reward-Free Exploration for Reinforcement Learningcited by 177
2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
2021Discovering and Achieving Goals via World Modelscited by 131
2021APS: Active Pretraining with Successor Featurescited by 109
2021Benchmarks for Deep Off-Policy Evaluationcited by 89
2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
2020Goal-GAN: Multimodal Trajectory Prediction Based on Goal Position Estimationcited by 57
2020Generalized Hindsight for Reinforcement Learningcited by 51
2021Collect & Infer -- a fresh look at data-efficient Reinforcement Learningcited by 17
2021Understanding the World Through Actioncited by 17
2020Batch Exploration with Examples for Scalable Robotic Reinforcement Learningcited by 14
2021Model Selection in Batch Policy Optimizationcited by 11
2021Efficient Self-Supervised Data Collection for Offline Robot Learningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 18; 2023: 19; 2024: 19; 2025: 25; 2026: 12.

93 papers.

Sort:

By area:

2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
2024Foundation Policies with Hilbert Representationscited by 89
2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 77
2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 60
2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
2022Masked Autoencoding for Scalable and Generalizable Decision Makingcited by 44
2022Evaluating Long-Term Memory in 3D Mazescited by 41

38 more

2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 40
2023Finetuning Offline World Models in the Real Worldcited by 39
2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
2022Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulationcited by 29
2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
2023The Generalization Gap in Offline Reinforcement Learningcited by 25
2022Boosting Offline Reinforcement Learning via Data Rebalancingcited by 22
2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
2023Accelerating Exploration with Unlabeled Prior Datacited by 21
2022General Intelligence Requires Rethinking Explorationcited by 20
2023When should we prefer Decision Transformers for Offline Reinforcement Learning?cited by 20
2023Survival Instinct in Offline Reinforcement Learningcited by 20
2025Temporal Difference Flowscited by 20
2025Fast Adaptation with Behavioral Foundation Modelscited by 20
2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
2023Distance Weighted Supervised Learning for Offline Interaction Datacited by 18
2024Proto Successor Measure: Representing the Behavior Space of an RL Agentcited by 17
2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learningcited by 17
2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraintscited by 13
2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12
2023A Survey of Demonstration Learningcited by 12
2023Blockwise Parallel Transformer for Large Context Modelscited by 12
2024Finer Behavioral Foundation Models via Auto-Regressive Features and Advantage Weightingcited by 12
2024Unsupervised-to-Online Reinforcement Learningcited by 11
2023Learning to View: Decision Transformers for Active Object Detectioncited by 10
2024Exclusively Penalized Q-learning for Offline Reinforcement Learningcited by 8
2024Do's and Don'ts: Learning Desirable Skills with Instruction Videoscited by 8
2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Biascited by 7
2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 7
2024Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Usecited by 7
2024RLZero: Direct Policy Inference from Language Without In-Domain Supervisioncited by 7
2025Intention-Conditioned Flow Occupancy Modelscited by 7
2026Hallucination in World Models is Predictable and Preventablecited by 7
2022Reliable Conditioning of Behavioral Cloning for Offline Reinforcement Learningcited by 6
2023Offline Reinforcement Learning with Imbalanced Datasetscited by 6

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way