mlsanityData to 7 Oct 2026

Hyperparameter Selection for Offline Reinforcement Learning

arXiv 2007.09055 ↗First posted 17 Jul 2020cs.LG cs.AI stat.MLcited by 113

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL purely from logged data) is an important avenue for deploying RL techniques in real-world scenarios. However, existing hyperparameter selection methods for offline RL break the offline assumption by evaluating policies corresponding to each hyperparameter setting in the environment. This online execution is often infeasible and hence undermines the main aim of offline RL. Therefore, in this work, we focus on textitoffline hyperparameter selection, i.e. methods for choosing the best policy from a set of many policies trained using different hyperparameters, given only logged data. Through large-scale empirical evaluation we show that: 1) offline RL algorithms are not robust to hyperparameter choices, 2) factors such as the offline RL algorithm and method for estimating Q values can have a big impact on hyperparameter selection, and 3) when we control those factors carefully, we can reliably rank policies across hyperparameter choices, and therefore choose policies which are close to the best policy in the set. Overall, our results present an optimistic view that offline hyperparameter selection is within reach, even in challenging tasks with pixel observations, high dimensional action spaces, and long horizon.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 49 references that are in the map. First posted on arXiv 17 Jul 2020.

Builds on

33 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2020Critic Regularized Regression26 Jun
  2. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  6. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 27

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  4. 2017Deep Reinforcement Learning that Matterscited by 727
  5. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  6. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  7. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  8. 2017Population Based Training of Neural Networkscited by 351
  9. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  10. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  11. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  12. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
15 more
  1. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  2. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  3. 2019Batch Policy Learning under Constraintscited by 242
  4. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  5. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  6. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  7. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  8. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  9. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  10. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 103
  11. 2018Neural probabilistic motor primitives for humanoid controlcited by 96
  12. 2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
  13. 2020POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learningcited by 36
  14. 2019Off-Policy Evaluation via Off-Policy Classificationcited by 32
  15. 2019Scaling data-driven robotics with reward sketching and batch reinforcement learningcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 34; 2022: 27; 2023: 20; 2024: 8; 2025: 10; 2026: 12.

113 papers.

Sort:

By area:

  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  2. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  4. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  5. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  6. 2021Offline RL Without Off-Policy Evaluationcited by 150
  7. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  8. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  9. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  10. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  11. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  12. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
38 more
  1. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 73
  2. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  3. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  4. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
  5. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  6. 2024Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrationscited by 51
  7. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50
  8. 2023Behavior Proximal Policy Optimizationcited by 43
  9. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  10. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  11. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 32
  12. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  13. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 31
  14. 2021Continuous Doubly Constrained Batch Reinforcement Learningcited by 27
  15. 2021Active Offline Policy Selectioncited by 26
  16. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matterscited by 24
  17. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
  18. 2022Policy-Adaptive Estimator Selection for Off-Policy Evaluationcited by 24
  19. 2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Modelscited by 24
  20. 2021UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planningcited by 23
  21. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
  22. 2020Semi-supervised reward learning for offline reinforcement learningcited by 22
  23. 2021On Instrumental Variable Regression for Deep Offline Policy Evaluationcited by 22
  24. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learningcited by 21
  25. 2021UAV-assisted Online Machine Learning over Multi-Tiered Networks: A Hierarchical Nested Personalized Federated Learning Approachcited by 20
  26. 2021Hyperparameter Selection for Imitation Learningcited by 17
  27. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
  28. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimizationcited by 16
  29. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learningcited by 13
  30. 2023Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluationcited by 13
  31. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Datacited by 12
  32. 2023A Survey of Demonstration Learningcited by 12
  33. 2021Model Selection in Batch Policy Optimizationcited by 11
  34. 2022User-Interactive Offline Reinforcement Learningcited by 11
  35. 2023Active Policy Improvement from Multiple Black-box Oraclescited by 11
  36. 2023ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learningcited by 11
  37. 2025RARE: Retrieval-Augmented Reasoning Modelingcited by 11
  38. 2025A Clean Slate for Offline Reinforcement Learningcited by 10

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  2. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  3. 2020Offline Reinforcement Learning Hands-On
  4. 2022Leveraging Offline Data in Online Reinforcement Learning
  5. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  6. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  7. 2020MOPO: Model-based Offline Policy Optimization
  8. 2021The Least Restriction for Offline Reinforcement Learning