mlsanityData to 7 Oct 2026

A Dataset Perspective on Offline Reinforcement Learning

arXiv 2111.04714 ↗First posted 8 Nov 2021cs.LG cs.AIcited by 27

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The application of Reinforcement Learning (RL) in real world environments can be expensive or risky due to sub-optimal policies during training. In Offline RL, this problem is avoided since interactions with an environment are prohibited. Policies are learned from a given dataset, which solely determines their performance. Despite this fact, how dataset characteristics influence Offline RL algorithms is still hardly investigated. The dataset characteristics are determined by the behavioral policy that samples this dataset. Therefore, we define characteristics of behavioral policies as exploratory for yielding high expected information in their interaction with the Markov Decision Process (MDP) and as exploitative for having high expected return. We implement two corresponding empirical measures for the datasets sampled by the behavioral policy in deterministic MDPs. The first empirical measure SACo is defined by the normalized unique state-action pairs and captures exploration. The second empirical measure TQ is defined by the normalized average trajectory return and captures exploitation. Empirical evaluations show the effectiveness of TQ and SACo. In large-scale experiments using our proposed measures, we show that the unconstrained off-policy Deep Q-Network family requires datasets with high SACo to find a good policy. Furthermore, experiments show that policy constraint algorithms perform well on datasets with high TQ and SACo. Finally, the experiments show, that purely dataset-constrained Behavioral Cloning performs competitively to the best Offline RL algorithms for datasets with high TQ.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 67 references that are in the map. First posted on arXiv 8 Nov 2021.

Builds on

35 of 67 references.

Closest ancestors · 11

Same direction, newest first

  1. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  2. 2021Regularized Behavior Value Estimation17 Mar
  3. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  4. 2020Offline Reinforcement Learning Hands-On29 Nov
  5. 2020Critic Regularized Regression26 Jun
  6. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  11. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 24

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2017Self-Normalizing Neural Networkscited by 591
  9. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  10. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
  11. 2019Challenges of Real-World Reinforcement Learningcited by 282
  12. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
12 more
  1. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  2. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
  3. 2019A review of domain adaptation without target labelscited by 154
  4. 2020MDP Homomorphic Networks: Group Symmetries in Reinforcement Learningcited by 133
  5. 2018An introduction to domain adaptation and transfer learningcited by 112
  6. 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 89
  7. 2020A Unifying View of Optimism in Episodic Reinforcement Learningcited by 50
  8. 2020Align-RUDDER: Learning From Few Demonstrations by Reward Redistributioncited by 35
  9. 2020Cross-Domain Few-Shot Learning by Representation Fusioncited by 32
  10. 2020Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDERcited by 19
  11. 2019Scaling data-driven robotics with reward sketching and batch reinforcement learningcited by 18
  12. 2021Collect & Infer -- a fresh look at data-efficient Reinforcement Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 9; 2025: 7; 2026: 7.

27 papers.

Sort:

By area:

  1. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  2. 2023Learning to Modulate pre-trained Models in RLcited by 28
  3. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  4. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 20
  5. 2024SymbolicAI: A framework for logic-based approaches combining generative models and solverscited by 15
  6. 2024A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Taskscited by 14
  7. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothnesscited by 12
  8. 2025Yes, Q-learning Helps Offline In-Context RLcited by 8
  9. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  10. 2024Improving Offline Reinforcement Learning with Inaccurate Simulatorscited by 4
  11. 2024Putting Data at the Centre of Offline Multi-Agent Reinforcement Learningcited by 4
  12. 2024Robust RL with LLM-Driven Data Synthesis and Policy Adaptation for Autonomous Drivingcited by 3
15 more
  1. 2024Offline Reinforcement Learning and Sequence Modeling for Downlink Link Adaptationcited by 3
  2. 2025Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhoodcited by 3
  3. 2024Contrastive Abstraction for Reinforcement Learningcited by 2
  4. 2025Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsitycited by 2
  5. 2024Integrating Domain Knowledge for handling Limited Data in Offline RLcited by 1
  6. 2026Epistemic Robust Offline Reinforcement Learningcited by 1
  7. 2025Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvementcited by 0
  8. 2025Generalizing Beyond Suboptimality: Offline Reinforcement Learning Learns Effective Scheduling through Random Solutionscited by 0
  9. 2025An Empirical Study on the Effectiveness of Incorporating Offline RL As Online RL Subroutinescited by 0
  10. 2026General Flexible f-divergence for Challenging Offline RL Datasets with Low Stochasticity and Diverse Behavior Policiescited by 0
  11. 2026Autonomous Drift Learning in Data Streams: A Unified Perspectivecited by 0
  12. 2026Trajectory-Level Data Augmentation for Offline Reinforcement Learningcited by 0
  13. 2026Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learningcited by 0
  14. 2026JumpStart Your Policy Learning with Lessons from 160,000 Training Runscited by 0
  15. 2026PORL: Pretrained Offline Reinforcement Learning for the Job Shop Scheduling Problemcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Measuring Data Quality for Dataset Selection in Offline Reinforcement Learning
  2. 2025Expert or not? assessing data quality in offline reinforcement learning
  3. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  4. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  5. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  6. 2021Interpretable performance analysis towards offline reinforcement learning: A dataset perspective
  7. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  8. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory