mlsanityData to 14 Oct 2026

Off-Policy Deep Reinforcement Learning without Exploration

arXiv 1812.02900 ↗First posted 7 Dec 2018cs.LG cs.AI stat.MLcited by 1,241

Not on the map. Why?

Cited by 770 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.

Key facts. Cited by 1,241 papers in the map (data through 14 Oct 2026). First posted on arXiv 7 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 8; 2020: 26; 2021: 144; 2022: 196; 2023: 234; 2024: 243; 2025: 196; 2026: 194.

1,241 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  7. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  9. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  10. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  12. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
38 more
  1. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  2. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  3. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  5. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  6. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  7. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  8. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
  9. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  10. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  11. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  12. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  13. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  14. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  15. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
  16. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  17. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  18. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 215
  19. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  20. 2022Multi-Game Decision Transformerscited by 214
  21. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  22. 2021Towards Causal Representation Learningcited by 209
  23. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 197
  24. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  25. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  26. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  27. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  28. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
  29. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 157
  30. 2021Offline RL Without Off-Policy Evaluationcited by 151
  31. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 149
  32. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  33. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  34. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  35. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  36. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  37. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
  38. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125

50 most cited of 1,241 All 1,241 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Critic Regularized Regression
  2. 2025An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithms
  3. 2024Efficient Off-Policy Learning for High-Dimensional Action Spaces
  4. 2019Batch Policy Learning under Constraints
  5. 2024Offline Reinforcement Learning from Datasets with Structured Non-Stationarity
  6. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  7. 2024The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations
  8. 2024Frugal Actor-Critic: Sample Efficient Off-Policy Deep Reinforcement Learning Using Unique Experiences