mlsanityData to 7 Oct 2026

d3rlpy: An Offline Deep Reinforcement Learning Library

arXiv 2111.03788 ↗First posted 6 Nov 2021cs.LG cs.AIcited by 84

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In this paper, we introduce d3rlpy, an open-sourced offline deep reinforcement learning (RL) library for Python. d3rlpy supports a set of offline deep RL algorithms as well as off-policy online algorithms via a fully documented plug-and-play API. To address a reproducibility issue, we conduct a large-scale benchmark with D4RL and Atari 2600 dataset to ensure implementation quality and provide experimental scripts and full tables of results. The d3rlpy source code can be found on GitHub: urlhttps://github.com/takuseno/d3rlpy.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 47 references that are in the map. First posted on arXiv 6 Nov 2021.

Builds on

35 of 47 references.

Closest ancestors · 8

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  3. 2020Critic Regularized Regression26 Jun
  4. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  11. 2017Deep Reinforcement Learning that Matterscited by 727
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
15 more
  1. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  3. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  4. 2018Learning to Drive in a Daycited by 266
  5. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  6. 2019Soft Actor-Critic for Discrete Action Settingscited by 161
  7. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
  8. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
  9. 2021RRL: Resnet as representation for Reinforcement Learningcited by 89
  10. 2017Time Limits in Reinforcement Learningcited by 72
  11. 2019ChainerRL: A Deep Reinforcement Learning Librarycited by 51
  12. 2020MushroomRL: Simplifying Reinforcement Learning Researchcited by 50
  13. 2020Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarkingcited by 18
  14. 2021SaLinA: Sequential Learning of Agentscited by 6
  15. 2020Interactive Visualization for Debugging RLcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 27; 2024: 18; 2025: 16; 2026: 16.

84 papers.

Sort:

By area:

  1. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  2. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  3. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  4. 2022Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Drivingcited by 71
  5. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
  6. 2023Train Offline, Test Online: A Real Robot Learning Benchmarkcited by 47
  7. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  8. 2022BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasetscited by 24
  9. 2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learningcited by 24
  10. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  11. 2024Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolfcited by 22
  12. 2022Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performancecited by 21
38 more
  1. 2023Replicable Reinforcement Learningcited by 21
  2. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 20
  3. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
  4. 2023Hierarchical Imitation Learning with Vector Quantized Modelscited by 15
  5. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learningcited by 13
  6. 2023Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluationcited by 13
  7. 2023ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learningcited by 11
  8. 2023Conservative State Value Estimation for Offline Reinforcement Learningcited by 9
  9. 2023Pearl: A Production-ready Reinforcement Learning Agentcited by 9
  10. 2024Learning from Sparse Offline Datasets via Conservative Density Estimationcited by 9
  11. 2023Improving Behavioural Cloning with Positive Unlabeled Learningcited by 8
  12. 2023Quasi-optimal Reinforcement Learning with Continuous Actionscited by 7
  13. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingcited by 7
  14. 2024TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agentscited by 7
  15. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluationcited by 6
  16. 2025Replicable Reinforcement Learning with Linear Function Approximationcited by 6
  17. 2023Identifying Expert Behavior in Offline Training Datasets Improves Behavioral Cloning of Robotic Manipulation Policiescited by 5
  18. 2023XuanCe: A Comprehensive and Unified Deep Reinforcement Learning Librarycited by 5
  19. 2025Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-trainingcited by 5
  20. 2024Multi-objective Cross-task Learning via Goal-conditioned GPT-based Decision Transformers for Surgical Robot Task Automationcited by 4
  21. 2023OER: Offline Experience Replay for Continual Offline Reinforcement Learningcited by 3
  22. 2023Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real Worldcited by 3
  23. 2023When is Offline Policy Selection Sample Efficient for Reinforcement Learning?cited by 3
  24. 2024Goal-Conditioned Reinforcement Learning from Sub-Optimal Data on Metric Spacescited by 3
  25. 2024Robust RL with LLM-Driven Data Synthesis and Policy Adaptation for Autonomous Drivingcited by 3
  26. 2025DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertaintycited by 3
  27. 2023Towards Real-World Applications of Personalized Anesthesia Using Policy Constraint Q Learning for Propofol Infusion Controlcited by 2
  28. 2024Solving Offline Reinforcement Learning with Decision Tree Regressioncited by 2
  29. 2025ImagineBench: Evaluating Reinforcement Learning with Large Language Model Rolloutscited by 2
  30. 2025Improving the Data-efficiency of Reinforcement Learning by Warm-starting with LLMcited by 2
  31. 2023HIPODE: Enhancing Offline Reinforcement Learning with High-Quality Synthetic Data from a Policy-Decoupled Approachcited by 1
  32. 2023Safe and Interpretable Estimation of Optimal Treatment Regimescited by 1
  33. 2023ToP-ToM: Trust-aware Robot Policy with Theory of Mindcited by 1
  34. 2024Dataset Clustering for Improved Offline Policy Learningcited by 1
  35. 2024Offline Goal-Conditioned Reinforcement Learning for Shape Control of Deformable Linear Objectscited by 1
  36. 2024Exploiting Symmetry in Dynamics for Model-Based Reinforcement Learning with Asymmetric Rewardscited by 1
  37. 2024Methodology for Interpretable Reinforcement Learning for Optimizing Mechanical Ventilationcited by 1
  38. 2024Oracle-Efficient Reinforcement Learning for Max Value Ensemblescited by 1

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library
  2. 2024D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
  3. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  4. 2025NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios
  5. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  6. 2025A Clean Slate for Offline Reinforcement Learning
  7. 2024State-Constrained Offline Reinforcement Learning
  8. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning