Data to 7 Oct 2026

Benchmarking Batch Deep Reinforcement Learning Algorithms

arXiv 1910.01708 ↗First posted 3 Oct 2019cs.LG cs.AI stat.MLcited by 108

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Widely-used deep reinforcement learning algorithms have been shown to fail in the batch setting--learning from a fixed data set without interaction with the environment. Following this result, there have been several papers showing reasonable performances under a variety of environments and batch settings. In this paper, we benchmark the performance of recent off-policy and batch reinforcement learning algorithms under unified settings on the Atari domain, with data generated by a single partially-trained behavioral policy. We find that under these conditions, many of these algorithms underperform DQN trained online with the same amount of data, as well as the partially-trained behavioral policy. To introduce a strong baseline, we adapt the Batch-Constrained Q-learning algorithm to a discrete-action setting, and show it outperforms all existing algorithms at this task.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 24 of its 49 references that are in the map. First posted on arXiv 3 Oct 2019.

Builds on

24 of 49 references.

Closest ancestors · 1

Same direction, newest first

2019An Optimistic Perspective on Offline Reinforcement Learning10 Jul

Other references · 23

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
2015Continuous control with deep reinforcement learningcited by 3,279
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
2016OpenAI Gymcited by 1,684
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2018Exploration by Random Network Distillationcited by 966
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
2017A Distributional Perspective on Reinforcement Learningcited by 743
2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632

11 more

2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
2018Horizon: Facebook's Open Source Applied Reinforcement Learning Platformcited by 48
2019Safe Policy Improvement with an Estimated Baseline Policycited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 7; 2021: 29; 2022: 19; 2023: 17; 2024: 16; 2025: 13; 2026: 6.

108 papers.

Sort:

By area:

2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
2021Offline RL Without Off-Policy Evaluationcited by 150
2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52

38 more

2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learningcited by 42
2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcarecited by 41
2022Modular Lifelong Reinforcement Learning via Neural Compositioncited by 39
2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learningcited by 33
2021Regularized Behavior Value Estimationcited by 32
2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 31
2022Safe Reinforcement Learning for Legged Locomotioncited by 29
2021Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancementcited by 28
2021A Dataset Perspective on Offline Reinforcement Learningcited by 27
2023The Generalization Gap in Offline Reinforcement Learningcited by 25
2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matterscited by 24
2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
2024Offline Model-Based Optimization via Policy-Guided Gradient Searchcited by 19
2022Semi-Markov Offline Reinforcement Learning for Healthcarecited by 18
2020DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPscited by 16
2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
2025Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Taskscited by 16
2020Benchmarking End-to-End Behavioural Cloning on Video Gamescited by 15
2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluationcited by 14
2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
2023ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learningcited by 11
2021False Correlation Reduction for Offline Reinforcement Learningcited by 9
2025Benchmarking Quantum Reinforcement Learningcited by 9
2023Improving Behavioural Cloning with Positive Unlabeled Learningcited by 8
2024EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systemscited by 8
2023Explaining RL Decisions with Trajectoriescited by 7
2024Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Usecited by 7
2020Offline Reinforcement Learning Hands-Oncited by 6
2020POPO: Pessimistic Offline Policy Optimizationcited by 5
2021RL-Scope: Cross-Stack Profiling for Deep Reinforcement Learning Workloadscited by 5
2022Offline Reinforcement Learning for Road Traffic Controlcited by 5
2023Identifying Expert Behavior in Offline Training Datasets Improves Behavioral Cloning of Robotic Manipulation Policiescited by 5
2024Scaling Vision-and-Language Navigation With Offline RLcited by 5
2024Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samplescited by 5
2021Batch Reinforcement Learning from Crowdscited by 4

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way