# Interpretable performance analysis towards offline reinforcement learning: A dataset perspective

arXiv: https://arxiv.org/abs/2105.05473 · HTML: https://mlsanity.com/paper/2105.05473/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 12 May 2021.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Builds on (most cited first, 13 of 13)

- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Safe Policy Improvement with Baseline Bootstrapping](https://mlsanity.com/paper/1712.06924/) (2017)
- [Benchmarking Batch Deep Reinforcement Learning Algorithms](https://mlsanity.com/paper/1910.01708/) (2019)
- [Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.01929/) (2016)
- [Deep Reinforcement Learning for Page-wise Recommendations](https://mlsanity.com/paper/1805.02343/) (2018)
- [BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning](https://mlsanity.com/paper/1910.12179/) (2019)
- [Robust Imitation of Diverse Behaviors](https://mlsanity.com/paper/1707.02747/) (2017)
- [Safe Policy Improvement by Minimizing Robust Baseline Regret](https://mlsanity.com/paper/1607.03842/) (2016)
- [Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application](https://mlsanity.com/paper/1803.00710/) (2018)

## Built on by (most cited first, 2 of 2)

- [Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation](https://mlsanity.com/paper/2109.08331/) (2021)
- [Frictional Q-Learning](https://mlsanity.com/paper/2509.19771/) (2025)

## Abstract (arXiv)

Offline reinforcement learning (RL) has increasingly become the focus of the artificial intelligent research due to its wide real-world applications where the collection of data may be difficult, time-consuming, or costly. In this paper, we first propose a two-fold taxonomy for existing offline RL algorithms from the perspective of exploration and exploitation tendency. Secondly, we derive the explicit expression of the upper bound of extrapolation error and explore the correlation between the performance of different types of algorithms and the distribution of actions under states. Specifically, we relax the strict assumption on the sufficiently large amount of state-action tuples. Accordingly, we provably explain why batch constrained Q-learning (BCQ) performs better than other existing techniques. Thirdly, after identifying the weakness of BCQ on dataset of low mean episode returns, we propose a modified variant based on top return selection mechanism, which is proved to be able to gain state-of-the-art performance on various datasets. Lastly, we create a benchmark platform on the Atari domain, entitled RL easy go (RLEG), at an estimated cost of more than 0.3 million dollars. We make it open-source for fair and comprehensive competitions between offline RL algorithms with complete datasets and checkpoints being provided.