mlsanityData to 7 Oct 2026

Measuring Data Quality for Dataset Selection in Offline Reinforcement Learning

arXiv 2111.13461 ↗First posted 26 Nov 2021cs.LGcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Recently developed offline reinforcement learning algorithms have made it possible to learn policies directly from pre-collected datasets, giving rise to a new dilemma for practitioners: Since the performance the algorithms are able to deliver depends greatly on the dataset that is presented to them, practitioners need to pick the right dataset among the available ones. This problem has so far not been discussed in the corresponding literature. We discuss ideas how to select promising datasets and propose three very simple indicators: Estimated relative return improvement (ERI) and estimated action stochasticity (EAS), as well as a combination of the two (COI), and empirically show that despite their simplicity they can be very effectively used for dataset selection.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 46 references that are in the map. First posted on arXiv 26 Nov 2021.

Builds on

26 of 46 references.

Closest ancestors · 13

Same direction, newest first

  1. 2021Behavior Constraining in Weight Space for Offline Reinforcement Learning12 Jul
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  3. 2021Risk-Averse Offline Reinforcement Learning10 Feb
  4. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  5. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
  6. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  7. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  8. 2020Critic Regularized Regression26 Jun
  9. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  10. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  11. 2020MOPO: Model-based Offline Policy Optimization27 May
  12. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
1 more
  1. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 13

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2020Model-Based Offline Planningcited by 126
  9. 2016Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networkscited by 49
  10. 2017A Benchmark Environment Motivated by Industrial Control Problemscited by 31
  11. 2019Outlier Detection for Improved Data Quality and Diversity in Dialog Systemscited by 28
  12. 2021Federated Noisy Client Learningcited by 13
1 more
  1. 2017Bayesian Alignments of Warped Multi-Output Gaussian Processescited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 1.

3 papers.

  1. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  2. 2023Offline Reinforcement Learning with Imbalanced Datasetscited by 6
  3. 2025Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Datasetcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2025Expert or not? assessing data quality in offline reinforcement learning
  2. 2021A Dataset Perspective on Offline Reinforcement Learning
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning
  4. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data
  5. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  6. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  7. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
  8. 2024Putting Data at the Centre of Offline Multi-Agent Reinforcement Learning