mlsanityData to 14 Oct 2026

Interpretable performance analysis towards offline reinforcement learning: A dataset perspective

arXiv 2105.05473 ↗First posted 12 May 2021cs.LG cs.AIcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) has increasingly become the focus of the artificial intelligent research due to its wide real-world applications where the collection of data may be difficult, time-consuming, or costly. In this paper, we first propose a two-fold taxonomy for existing offline RL algorithms from the perspective of exploration and exploitation tendency. Secondly, we derive the explicit expression of the upper bound of extrapolation error and explore the correlation between the performance of different types of algorithms and the distribution of actions under states. Specifically, we relax the strict assumption on the sufficiently large amount of state-action tuples. Accordingly, we provably explain why batch constrained Q-learning (BCQ) performs better than other existing techniques. Thirdly, after identifying the weakness of BCQ on dataset of low mean episode returns, we propose a modified variant based on top return selection mechanism, which is proved to be able to gain state-of-the-art performance on various datasets. Lastly, we create a benchmark platform on the Atari domain, entitled RL easy go (RLEG), at an estimated cost of more than 0.3 million dollars. We make it open-source for fair and comprehensive competitions between offline RL algorithms with complete datasets and checkpoints being provided.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 13 of its 18 references that are in the map. First posted on arXiv 12 May 2021.

Builds on

13 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 12

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  3. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  4. 2016Deep Exploration via Bootstrapped DQNcited by 527
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  6. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  7. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  8. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 100
  9. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  10. 2017Robust Imitation of Diverse Behaviorscited by 86
  11. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 53
  12. 2018Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Applicationcited by 34

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 0; 2023: 0; 2024: 0; 2025: 1.

2 papers.

  1. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
  2. 2025Frictional Q-Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A Dataset Perspective on Offline Reinforcement Learning
  2. 2019An Optimistic Perspective on Offline Reinforcement Learning
  3. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  4. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  5. 2021The Least Restriction for Offline Reinforcement Learning
  6. 2021Continuous Doubly Constrained Batch Reinforcement Learning
  7. 2021A Minimalist Approach to Offline Reinforcement Learning
  8. 2024State-Constrained Offline Reinforcement Learning