mlsanityData to 14 Oct 2026

When is Offline Policy Selection Sample Efficient for Reinforcement Learning?

arXiv 2312.02355 ↗First posted 4 Dec 2023cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning algorithms often require careful hyperparameter tuning. Before deployment, we need to select amongst a set of candidate policies. However, there is limited understanding about the fundamental limits of this offline policy selection (OPS) problem. In this work we provide clarity on when sample efficient OPS is possible, primarily by connecting OPS to off-policy policy evaluation (OPE) and Bellman error (BE) estimation. We first show a hardness result, that in the worst case, OPS is just as hard as OPE, by proving a reduction of OPE to OPS. As a result, no OPS method can be more sample efficient than OPE in the worst case. We then connect BE estimation to the OPS problem, showing how BE can be used as a tool for OPS. While BE-based methods generally require stronger requirements than OPE, when those conditions are met they can be more sample efficient. Building on this insight, we propose a BE method for OPS, called Identifiable BE Selection (IBES), that has a straightforward method for selecting its own hyperparameters. We conclude with an empirical study comparing OPE and IBES, and by showing the difficulty of OPS on an offline Atari benchmark dataset.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 54 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

37 of 54 references.

Closest ancestors · 19

Same direction, newest first

  1. 2023Revisiting Bellman Errors for Offline Model Selection31 Jan
  2. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning3 Nov
  3. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data16 Oct
  4. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  5. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  6. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning26 Oct
  7. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  8. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
  9. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  10. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  11. 2021Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settings13 May
  12. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning25 Mar
7 more
  1. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  2. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  3. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 18

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  3. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  4. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  5. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  6. 2019Batch Policy Learning under Constraintscited by 244
  7. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  8. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  9. 2021Conservative Objective Models for Effective Offline Model-Based Optimizationcited by 88
  10. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  11. 2019Model selection for contextual banditscited by 60
  12. 2020Adaptive Estimator Selection for Off-Policy Evaluationcited by 38
6 more
  1. 2020Offline Policy Selection under Uncertaintycited by 31
  2. 2022Data-Driven Offline Decision-Making via Invariant Representation Learningcited by 31
  3. 2020Online Model Selection for Reinforcement Learning with Function Approximationcited by 28
  4. 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
  5. 2021Model Selection in Batch Policy Optimizationcited by 11
  6. 2021The Curse of Passive Data Collection in Batch Reinforcement Learningcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selectioncited by 15
  2. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocolcited by 6
  3. 2025STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  2. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
  3. 2026A Unifying View of Coverage in Linear Off-Policy Evaluation
  4. 2021Active Offline Policy Selection
  5. 2026Active Offline-to-Online Reinforcement Learning
  6. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  7. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability
  8. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples