mlsanityData to 7 Oct 2026

An Optimistic Perspective on Offline Reinforcement Learning

arXiv 1907.04543 ↗First posted 10 Jul 2019cs.LG cs.AI stat.MLcited by 37

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Off-policy reinforcement learning (RL) using a fixed offline dataset of logged interactions is an important consideration in real world applications. This paper studies offline RL using the DQN replay dataset comprising the entire replay experience of a DQN agent on 60 Atari 2600 games. We demonstrate that recent off-policy deep RL algorithms, even when trained solely on this fixed dataset, outperform the fully trained DQN agent. To enhance generalization in the offline setting, we present Random Ensemble Mixture (REM), a robust Q-learning algorithm that enforces optimal Bellman consistency on random convex combinations of multiple Q-value estimates. Offline REM trained on the DQN replay dataset surpasses strong RL baselines. Ablation studies highlight the role of offline dataset size and diversity as well as the algorithm choice in our positive results. Overall, the results here present an optimistic view that robust RL algorithms trained on sufficiently large and diverse offline datasets can lead to high quality policies. The DQN replay dataset can serve as an offline RL benchmark and is open-sourced.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 62 references that are in the map. First posted on arXiv 10 Jul 2019.

Builds on

32 of 62 references.

References · 32

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2015Prioritized Experience Replaycited by 1,070
  7. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  10. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  11. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  12. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
20 more
  1. 2018Learning Dexterous In-Hand Manipulationcited by 643
  2. 2016Deep Exploration via Bootstrapped DQNcited by 518
  3. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  4. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  5. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  6. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  7. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  8. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  9. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  10. 2017Noisy Networks for Explorationcited by 262
  11. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 228
  12. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  13. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
  14. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  15. 2017A Deeper Look at Experience Replaycited by 104
  16. 2017Towards Generalization and Simplicity in Continuous Controlcited by 65
  17. 2019A Geometric Perspective on Optimal Representations for Reinforcement Learningcited by 54
  18. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  19. 2019Off-Policy Evaluation via Off-Policy Classificationcited by 32
  20. 2019Distributional reinforcement learning with linear function approximationcited by 15

Built on by

Cited by year

Citing papers per year: 2019: 6; 2020: 9; 2021: 6; 2022: 6; 2023: 6; 2024: 0; 2025: 4.

37 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  4. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  5. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  6. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  7. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
  8. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  9. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  10. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  11. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learningcited by 42
  12. 2023Finetuning Offline World Models in the Real Worldcited by 39
25 more
  1. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  2. 2021Regularized Behavior Value Estimationcited by 32
  3. 2019Scaling data-driven robotics with reward sketching and batch reinforcement learningcited by 18
  4. 2022Dungeons and Data: A Large-Scale NetHack Datasetcited by 16
  5. 2025Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robotscited by 15
  6. 2023Learning from Visual Observation via Offline Pretrained State-to-Go Transformercited by 14
  7. 2022Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learningcited by 13
  8. 2021PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulatorscited by 10
  9. 2025A Clean Slate for Offline Reinforcement Learningcited by 10
  10. 2022Offline Reinforcement Learning for Mobile Notificationscited by 8
  11. 2022ASPiRe:Adaptive Skill Priors for Reinforcement Learningcited by 7
  12. 2019ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrationscited by 5
  13. 2020POPO: Pessimistic Offline Policy Optimizationcited by 5
  14. 2021Q-Value Weighted Regression: Reinforcement Learning with Limited Datacited by 4
  15. 2021Maximum Entropy Reinforcement Learning with Mixture Policiescited by 4
  16. 2021Importance of Empirical Sample Complexity Analysis for Offline Reinforcement Learningcited by 3
  17. 2023BCQQ: Batch-Constraint Quantum Q-Learning with Cyclic Data Re-uploadingcited by 3
  18. 2025Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsitycited by 2
  19. 2019Multi-task Batch Reinforcement Learning with Metric Learningcited by 1
  20. 2022A Maintenance Planning Framework using Online and Offline Deep Reinforcement Learningcited by 0
  21. 2022CNT (Conditioning on Noisy Targets): A new Algorithm for Leveraging Top-Down Feedbackcited by 0
  22. 2023Dissecting Continual Learning a Structural and Data Analysiscited by 0
  23. 2023Risk-Sensitive and Robust Model-Based Reinforcement Learning and Planningcited by 0
  24. 2023Investigating the Edge of Stability Phenomenon in Reinforcement Learningcited by 0
  25. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  2. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
  3. 2021Interpretable performance analysis towards offline reinforcement learning: A dataset perspective
  4. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
  5. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  6. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
  7. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  8. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes