mlsanityData to 14 Oct 2026

Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning

arXiv 2110.14000 ↗First posted 26 Oct 2021cs.LG cs.AI stat.MLcited by 32

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

How to select between policies and value functions produced by different training algorithms in offline reinforcement learning (RL) -- which is crucial for hyperpa-rameter tuning -- is an important open question. Existing approaches based on off-policy evaluation (OPE) often require additional function approximation and hence hyperparameters, creating a chicken-and-egg situation. In this paper, we design hyperparameter-free algorithms for policy selection based on BVFT [XJ21], a recent theoretical advance in value-function selection, and demonstrate their effectiveness in discrete-action benchmarks such as Atari. To address performance degradation due to poor critics in continuous-action domains, we further combine BVFT with OPE to get the best of both worlds, and obtain a hyperparameter-tuning method for Q-function based OPE with theoretical guarantees as a side product.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 39 references that are in the map. First posted on arXiv 26 Oct 2021.

Builds on

25 of 39 references.

Closest ancestors · 12

Same direction, newest first

  1. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  3. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency5 Feb
  4. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  5. 2020Batch Value-function Approximation with Only Realizability11 Aug
  6. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  7. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 13

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  2. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  3. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  4. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  5. 2019Batch Policy Learning under Constraintscited by 244
  6. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  7. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  8. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  9. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  10. 2019Off-Policy Policy Gradient with State Distribution Correctioncited by 43
  11. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  12. 2020Offline Policy Selection under Uncertaintycited by 31
1 more
  1. 2020Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 13; 2023: 9; 2024: 2; 2025: 3; 2026: 3.

32 papers.

Sort:

By area:

  1. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  2. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  3. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  4. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
  5. 2022Robust Reinforcement Learning using Offline Datacited by 96
  6. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 66
  7. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
  8. 2023Behavior Proximal Policy Optimizationcited by 43
  9. 2022Policy-Adaptive Estimator Selection for Off-Policy Evaluationcited by 24
  10. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targetscited by 22
  11. 2023Efficient Policy Evaluation with Offline Data Informed Behavior Policy Designcited by 14
  12. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learningcited by 13
20 more
  1. 2023Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluationcited by 13
  2. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Datacited by 12
  3. 2021Model Selection in Batch Policy Optimizationcited by 11
  4. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
  5. 2022Beyond the Return: Off-policy Function Estimation under User-specified Error-measuring Distributionscited by 8
  6. 2022Reinforcement Learning in Practice: Opportunities and Challengescited by 6
  7. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluationcited by 6
  8. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocolcited by 6
  9. 2023Revisiting Bellman Errors for Offline Model Selectioncited by 5
  10. 2022Incorporating Explicit Uncertainty Estimates into Deep Offline Reinforcement Learningcited by 4
  11. 2023OER: Offline Experience Replay for Continual Offline Reinforcement Learningcited by 4
  12. 2023When is Offline Policy Selection Sample Efficient for Reinforcement Learning?cited by 3
  13. 2024OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimatorscited by 3
  14. 2021Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learningcited by 2
  15. 2022Hyperparameter Selection Methods for Fitted Q-Evaluation with Error Guaranteecited by 1
  16. 2026A Unifying View of Coverage in Linear Off-Policy Evaluationcited by 1
  17. 2026SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Datacited by 1
  18. 2026A More Accurate Algorithm Comparison through A/B Testing using Offline Evaluation Methodscited by 1
  19. 2024Off-Policy Selection for Initiating Human-Centric Experimental Designcited by 0
  20. 2025Selecting Belief-State Approximations in Simulators with Latent Statescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Clean Slate for Offline Reinforcement Learning
  2. 2021Offline RL Without Off-Policy Evaluation
  3. 2021Interpretable performance analysis towards offline reinforcement learning: A dataset perspective
  4. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  5. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  6. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning
  7. 2025Fully Offline Reinforcement Learning
  8. 2022Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learning