mlsanityData to 7 Oct 2026

Data Valuation for Offline Reinforcement Learning

arXiv 2205.09550 ↗First posted 19 May 2022cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The success of deep reinforcement learning (DRL) hinges on the availability of training data, which is typically obtained via a large number of environment interactions. In many real-world scenarios, costs and risks are associated with gathering these data. The field of offline reinforcement learning addresses these issues through outsourcing the collection of data to a domain expert or a carefully monitored program and subsequently searching for a batch-constrained optimal policy. With the emergence of data markets, an alternative to constructing a dataset in-house is to purchase external data. However, while state-of-the-art offline reinforcement learning approaches have shown a lot of promise, they currently rely on carefully constructed datasets that are well aligned with the intended target domains. This raises questions regarding the transferability and robustness of an offline reinforcement learning agent trained on externally acquired data. In this paper, we empirically evaluate the ability of the current state-of-the-art offline reinforcement learning approaches to coping with the source-target domain mismatch within two MuJoCo environments, finding that current state-of-the-art offline reinforcement learning algorithms underperform in the target domain. To address this, we propose data valuation for offline reinforcement learning (DVORL), which allows us to identify relevant and high-quality transitions, improving the performance and transferability of policies learned by offline reinforcement learning algorithms. The results show that our method outperforms offline reinforcement learning baselines on two MuJoCo environments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 37 references that are in the map. First posted on arXiv 19 May 2022.

Builds on

25 of 37 references.

Closest ancestors · 5

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  5. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 20

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2016OpenAI Gymcited by 1,684
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2015Prioritized Experience Replaycited by 1,070
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 617
  7. 2018Distributed Prioritized Experience Replaycited by 260
  8. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  9. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  10. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  11. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  12. 2017Preparing for the Unknown: Learning a Universal Policy with Online System Identificationcited by 129
8 more
  1. 2019Data Valuation using Reinforcement Learningcited by 115
  2. 2020Neuron Shapley: Discovering the Responsible Neuronscited by 98
  3. 2020A Distributional Framework for Data Valuationcited by 79
  4. 2021Safe Multi-Agent Reinforcement Learning via Shieldingcited by 60
  5. 2019Safe Reinforcement Learning on Autonomous Vehiclescited by 48
  6. 2013Are all training examples equally valuable?cited by 14
  7. 2021Training Data Subset Selection for Regression with Controlled Generalization Errorcited by 14
  8. 2022SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisitioncited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Expert or not? assessing data quality in offline reinforcement learning
  2. 2025Enhancing Offline Reinforcement Learning with Curriculum Learning-Based Trajectory Valuation
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning
  4. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
  6. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  7. 2024Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samples
  8. 2021A Dataset Perspective on Offline Reinforcement Learning