mlsanityData to 7 Oct 2026

Offline RL Without Off-Policy Evaluation

arXiv 2106.08909 ↗First posted 16 Jun 2021cs.LG stat.MLcited by 150

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Most prior approaches to offline reinforcement learning (RL) have taken an iterative actor-critic approach involving off-policy evaluation. In this paper we show that simply doing one step of constrained/regularized policy improvement using an on-policy Q estimate of the behavior policy performs surprisingly well. This one-step algorithm beats the previously reported results of iterative algorithms on a large portion of the D4RL benchmark. The one-step baseline achieves this strong performance while being notably simpler and more robust to hyperparameters than previously proposed iterative algorithms. We argue that the relatively poor performance of iterative approaches is a result of the high variance inherent in doing off-policy evaluation and magnified by the repeated optimization of policies against those estimates. In addition, we hypothesize that the strong performance of the one-step algorithm is due to a combination of favorable structure in the environment and behavior policy.

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 58 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

36 of 58 references.

Closest ancestors · 13

Same direction, newest first

  1. 2021Regularized Behavior Value Estimation17 Mar
  2. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  3. 2021Instabilities of Offline RL with Pre-Trained Neural Representation8 Mar
  4. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  5. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  6. 2020Critic Regularized Regression26 Jun
  7. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  11. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  12. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
1 more
  1. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2017Constrained Policy Optimizationcited by 928
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  11. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  12. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
11 more
  1. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  2. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  3. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  4. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  5. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  6. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  7. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  8. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  9. 2020A Variant of the Wang-Foster-Kakade Lower Bound for the Discounted Settingcited by 24
  10. 2019Safe Policy Improvement with Soft Baseline Bootstrappingcited by 22
  11. 2019Imitation-Regularized Offline Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 26; 2023: 42; 2024: 33; 2025: 20; 2026: 22.

150 papers.

Sort:

By area:

  1. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  2. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  3. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  5. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  6. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  7. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  8. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  9. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  10. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  11. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  12. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
38 more
  1. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  2. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  3. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
  4. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  5. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  6. 2024Diffusion Actor-Critic with Entropy Regulatorcited by 81
  7. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 73
  8. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  9. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
  10. 2025Horizon Reduction Makes RL Scalablecited by 55
  11. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  12. 2023Behavior Proximal Policy Optimizationcited by 43
  13. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
  14. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learningcited by 40
  15. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  16. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitchingcited by 35
  17. 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
  18. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
  19. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  20. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  21. 2023Bridging RL Theory and Practice with the Effective Horizoncited by 31
  22. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  23. 2022Sustainable Online Reinforcement Learning for Auto-biddingcited by 28
  24. 2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
  25. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matterscited by 24
  26. 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
  27. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  28. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
  29. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weightingcited by 22
  30. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targetscited by 22
  31. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RLcited by 21
  32. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  33. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  34. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuningcited by 20
  35. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
  36. 2023Supported Trust Region Optimization for Offline Reinforcement Learningcited by 20
  37. 2024AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimizationcited by 19
  38. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18

50 most cited of 150 All 150 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
  2. 2021A Minimalist Approach to Offline Reinforcement Learning
  3. 2025A Clean Slate for Offline Reinforcement Learning
  4. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  5. 2020MOReL : Model-Based Offline Reinforcement Learning
  6. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  7. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation
  8. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL