mlsanityData to 14 Oct 2026

Model Selection in Batch Policy Optimization

arXiv 2112.12320 ↗First posted 23 Dec 2021cs.LG stat.MLcited by 11

Not on the map. Why?

Abstract (arXiv)

We study the problem of model selection in batch policy optimization: given a fixed, partial-feedback dataset and M model classes, learn a policy with performance that is competitive with the policy derived from the best model class. We formalize the problem in the contextual bandit setting with linear model classes by identifying three sources of error that any model selection algorithm should optimally trade-off in order to be competitive: (1) approximation error, (2) statistical complexity, and (3) coverage. The first two sources are common in model selection for supervised learning, where optimally trading-off these properties is well-studied. In contrast, the third source is unique to batch policy optimization and is due to dataset shift inherent to the setting. We first show that no batch policy optimization algorithm can achieve a guarantee addressing all three simultaneously, revealing a stark contrast between difficulties in batch policy optimization and the positive results available in supervised learning. Despite this negative result, we show that relaxing any one of the three error sources enables the design of algorithms achieving near-oracle inequalities for the remaining two. We conclude with experiments demonstrating the efficacy of these algorithms.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 31 of its 52 references that are in the map. First posted on arXiv 23 Dec 2021.

Builds on

31 of 52 references.

References · 31

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  3. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  4. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  5. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  6. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  7. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  8. 2020Beyond UCB: Optimal and Efficient Contextual Bandits with Regression Oraclescited by 197
  9. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  10. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
  11. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  12. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
19 more
  1. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 110
  2. 2020Batch Value-function Approximation with Only Realizabilitycited by 104
  3. 2016Corralling a Band of Bandit Algorithmscited by 100
  4. 2019Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensemblescited by 100
  5. 2020Model Selection in Contextual Stochastic Bandit Problemscited by 71
  6. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 69
  7. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  8. 2019Model selection for contextual banditscited by 60
  9. 2020Adaptive Estimator Selection for Off-Policy Evaluationcited by 38
  10. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  11. 2020Offline Policy Selection under Uncertaintycited by 31
  12. 2021Leveraging Good Representations in Linear Contextual Banditscited by 29
  13. 2019OSOM: A simultaneously optimal algorithm for multi-armed and linear contextual banditscited by 28
  14. 2020Online Model Selection for Reinforcement Learning with Function Approximationcited by 28
  15. 2021On the Optimality of Batch Policy Optimization Algorithmscited by 28
  16. 2020Problem-Complexity Adaptive Model Selection for Stochastic Linear Banditscited by 26
  17. 2021Provably Efficient Representation Selection in Low-rank Markov Decision Processes: From Online to Offline RLcited by 12
  18. 2013Bounded regret in stochastic multi-armed banditscited by 10
  19. 2021Universal and data-adaptive algorithms for model selection in linear contextual banditscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 3; 2024: 2; 2025: 1; 2026: 1.

11 papers.

  1. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
  2. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  3. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learningcited by 13
  4. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Datacited by 12
  5. 2022Bellman Residual Orthogonalization for Offline Reinforcement Learningcited by 10
  6. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocolcited by 6
  7. 2023When is Offline Policy Selection Sample Efficient for Reinforcement Learning?cited by 3
  8. 2024Cross-Validated Off-Policy Evaluationcited by 3
  9. 2023Estimating Optimal Policy Value in General Linear Contextual Banditscited by 0
  10. 2024Off-Policy Selection for Initiating Human-Centric Experimental Designcited by 0
  11. 2026Unlearning Offline Stochastic Multi-Armed Banditscited by 0