mlsanityData to 14 Oct 2026

The Difficulty of Passive Learning in Deep Reinforcement Learning

arXiv 2110.14020 ↗First posted 26 Oct 2021cs.LG cs.AIcited by 50

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Learning to act from observational data without active environmental interaction is a well-known challenge in Reinforcement Learning (RL). Recent approaches involve constraints on the learned policy or conservative updates, preventing strong deviations from the state-action distribution of the dataset. Although these methods are evaluated using non-linear function approximation, theoretical justifications are mostly limited to the tabular or linear cases. Given the impressive results of deep reinforcement learning, we argue for a need to more clearly understand the challenges in this setting. In the vein of Held & Hein's classic 1963 experiment, we propose the "tandem learning" experimental paradigm which facilitates our empirical analysis of the difficulties in offline reinforcement learning. We identify function approximation in conjunction with fixed data distributions as the strongest factors, thereby extending but also challenging hypotheses stated in past work. Our results provide relevant insights for offline deep reinforcement learning, while also shedding new light on phenomena observed in the online case of learning control.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 52 references that are in the map. First posted on arXiv 26 Oct 2021.

Builds on

32 of 52 references.

Closest ancestors · 13

Same direction, newest first

  1. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  2. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model13 Apr
  3. 2021Instabilities of Offline RL with Pre-Trained Neural Representation8 Mar
  4. 2021Continuous Doubly Constrained Batch Reinforcement Learning18 Feb
  5. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  6. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  7. 2020Critic Regularized Regression26 Jun
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOPO: Model-based Offline Policy Optimization27 May
  10. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  12. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
1 more
  1. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  8. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  9. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  10. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  11. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  12. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
7 more
  1. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  2. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  3. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  4. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  5. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  6. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  7. 2020Interference and Generalization in Temporal Difference Learningcited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 13; 2024: 18; 2025: 8; 2026: 3.

50 papers.

Sort:

By area:

  1. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  2. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  3. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 78
  4. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 65
  5. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 63
  6. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 39
  7. 2025Flow-Based Policy for Online Reinforcement Learningcited by 37
  8. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  9. 2023Cross-Domain Policy Adaptation via Value-Guided Data Filteringcited by 31
  10. 2022The Phenomenon of Policy Churncited by 28
  11. 2023The Generalization Gap in Offline Reinforcement Learningcited by 26
  12. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
38 more
  1. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  2. 2023Passive learning of active causal strategies in agents and language modelscited by 21
  3. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
  4. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  5. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 17
  6. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  7. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 14
  8. 2022Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networkscited by 13
  9. 2023Bad Habits: Policy Confounding and Out-of-Trajectory Generalization in RLcited by 10
  10. 2023Conservative State Value Estimation for Offline Reinforcement Learningcited by 9
  11. 2023Can Agents Run Relay Race with Strangers? Generalization of RL to Out-of-Distribution Trajectoriescited by 9
  12. 2024BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignmentcited by 9
  13. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Biascited by 7
  14. 2023Understanding when Dynamics-Invariant Data Augmentations Benefit Model-Free Reinforcement Learning Updatescited by 7
  15. 2022CUP: Critic-Guided Policy Reusecited by 6
  16. 2023Synergizing Quality-Diversity with Descriptor-Conditioned Reinforcement Learningcited by 6
  17. 2024Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learningcited by 6
  18. 2024The Curse of Diversity in Ensemble-Based Explorationcited by 6
  19. 2025Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learningcited by 6
  20. 2025Compute-Optimal Scaling for Value-Based Deep RLcited by 6
  21. 2024CALE: Continuous Arcade Learning Environmentcited by 5
  22. 2025Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RLcited by 5
  23. 2024Predictive representations: building blocks of intelligencecited by 4
  24. 2025Optimistic critics can empower small actorscited by 4
  25. 2025Bridging the Performance Gap Between Target-Free and Target-Based Reinforcement Learningcited by 4
  26. 2024Adaptive Q-Network: On-the-fly Target Selection for Deep Reinforcement Learningcited by 3
  27. 2026On the Role of Computation in Reinforcement Learningcited by 3
  28. 2023Scaling Goal-based Exploration via Pruning Proto-goalscited by 2
  29. 2026Can Vision Language Models Learn Intuitive Physics from Interaction?cited by 2
  30. 2023Knowledge Transfer from Teachers to Learners in Growing-Batch Reinforcement Learningcited by 1
  31. 2024Learning Off-policy with Model-based Intrinsic Motivation For Active Online Explorationcited by 1
  32. 2024Aligning Agents like Large Language Modelscited by 1
  33. 2025Is Exploration or Optimization the Problem for Deep Reinforcement Learning?cited by 1
  34. 2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategiescited by 1
  35. 2026Learning to Learn from Language Feedback with Social Meta-Learningcited by 1
  36. 2022The Pump Scheduling Problem: A Real-World Scenario for Reinforcement Learningcited by 0
  37. 2022Efficient Reinforcement Learning Through Trajectory Generationcited by 0
  38. 2024Dynamic Learning Rate for Deep Reinforcement Learning: A Bandit Approachcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  2. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  4. 2021A Dataset Perspective on Offline Reinforcement Learning
  5. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  6. 2024Simple Ingredients for Offline Reinforcement Learning
  7. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  8. 2026Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies