mlsanityData to 7 Oct 2026

Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol

arXiv 2502.08021 ↗First posted 11 Feb 2025cs.LG cs.AI stat.MLcited by 6

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Holdout validation and hyperparameter tuning from data is a long-standing problem in offline reinforcement learning (RL). A standard framework is to use off-policy evaluation (OPE) methods to evaluate and select the policies, but OPE either incurs exponential variance (e.g., importance sampling) or has hyperparameters on their own (e.g., FQE and model-based). We focus on hyperparameter tuning for OPE itself, which is even more under-investigated. Concretely, we select among candidate value functions ("model-free") or dynamics ("model-based") to best assess the performance of a target policy. Concretely, we select among candidate value functions (``model-free'') or dynamics models (``model-based'') to best assess the performance of a target policy. We develop: (1) new model-free and model-based selectors with theoretical guarantees, and (2) a new experimental protocol for empirically evaluating them. Compared to the model-free protocol in prior works, our new protocol allows for more stable generation and better control of candidate value functions in an optimization-free manner, and evaluation of model-free and model-based methods alike. We exemplify the protocol on Gym-Hopper, and find that our new model-free selector, LSTD-Tournament, demonstrates promising empirical performance.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 66 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

43 of 66 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Offline Reinforcement Learning: Role of State Aggregation and Trajectory Data25 Mar
  2. 2023When is Offline Policy Selection Sample Efficient for Reinforcement Learning?4 Dec
  3. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluation30 Nov
  4. 2023Revisiting Bellman Errors for Offline Model Selection31 Jan
  5. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning3 Nov
  6. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data16 Oct
  7. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  8. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning26 Oct
  9. 2021Representation Learning for Online and Offline RL in Low-rank MDPs9 Oct
  10. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
  11. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  12. 2020Batch Value-function Approximation with Only Realizability11 Aug
3 more
  1. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 28

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2018World Modelscited by 1,444
  3. 2023Mastering Diverse Domains through World Modelscited by 1,193
  4. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  5. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  6. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  7. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  8. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  9. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  10. 2019Batch Policy Learning under Constraintscited by 242
  11. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  12. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
16 more
  1. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  2. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  3. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  4. 2020Adaptive Estimator Selection for Off-Policy Evaluationcited by 38
  5. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 31
  6. 2020A Variant of the Wang-Foster-Kakade Lower Bound for the Discounted Settingcited by 24
  7. 2022Policy-Adaptive Estimator Selection for Off-Policy Evaluationcited by 24
  8. 2021On Instrumental Variable Regression for Deep Offline Policy Evaluationcited by 22
  9. 2021Minimax Model Learningcited by 16
  10. 2021Model Selection in Batch Policy Optimizationcited by 11
  11. 2022A Complete Characterization of Linear Estimators for Offline Policy Evaluationcited by 7
  12. 2024A Note on Loss Functions and Error Compounding in Model-based Reinforcement Learningcited by 7
  13. 2018High-confidence error estimates for learned value functionscited by 4
  14. 2024Mitigating Covariate Shift in Misspecified Regression with Applications to Reinforcement Learningcited by 4
  15. 2023The Optimal Approximation Factors in Misspecified Off-Policy Value Function Estimationcited by 3
  16. 2024Reinforcement Learning under Latent Dynamics: Toward Statistical and Algorithmic Modularitycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
  2. 2026A Unifying View of Coverage in Linear Off-Policy Evaluationcited by 1
  3. 2025Selecting Belief-State Approximations in Simulators with Latent Statescited by 0
  4. 2025Improved Training Mechanism for Reinforcement Learning via Online Model Selectioncited by 0
  5. 2026Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matchingcited by 0
  6. 2026Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Clean Slate for Offline Reinforcement Learning
  2. 2020MOPO: Model-based Offline Policy Optimization
  3. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective
  4. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  5. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  6. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  7. 2026Active Offline-to-Online Reinforcement Learning
  8. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization