mlsanityData to 7 Oct 2026

Optimal and Adaptive Off-policy Evaluation in Contextual Bandits

arXiv 1612.01205 ↗First posted 4 Dec 2016stat.ML cs.LGcited by 111

Not on the map. Why?

Abstract (arXiv)

We study the off-policy evaluation problem---estimating the value of a target policy using data collected by another policy---under the contextual bandit model. We consider the general (agnostic) setting without access to a consistent model of rewards and establish a minimax lower bound on the mean squared error (MSE). The bound is matched up to constants by the inverse propensity scoring (IPS) and doubly robust (DR) estimators. This highlights the difficulty of the agnostic contextual setting, in contrast with multi-armed bandits and contextual bandits with access to a consistent reward model, where IPS is suboptimal. We then propose the SWITCH estimator, which can use an existing reward model (not necessarily consistent) to achieve a better bias-variance tradeoff than IPS and DR. We prove an upper bound on its MSE and demonstrate its benefits empirically on a diverse collection of data sets, often outperforming prior work by orders of magnitude.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 27; 2022: 16; 2023: 23; 2024: 19; 2025: 7; 2026: 17.

111 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
  3. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  4. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  5. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  6. 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 53
  7. 2021Statistical Inference with M-Estimators on Adaptively Collected Datacited by 40
  8. 2021Off-Policy Evaluation via Adaptive Weighting with Data from Contextual Banditscited by 40
  9. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  10. 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
  11. 2021Post-Contextual-Bandit Inferencecited by 32
  12. 2021Evaluating the Robustness of Off-Policy Evaluationcited by 32
38 more
  1. 2021Off-Policy Risk Assessment in Contextual Banditscited by 29
  2. 2021Estimating Average Treatment Effects via Orthogonal Regularizationcited by 25
  3. 2022Conformal Off-Policy Prediction in Contextual Banditscited by 25
  4. 2022Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimationcited by 25
  5. 2021Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorabilitycited by 24
  6. 2022Policy-Adaptive Estimator Selection for Off-Policy Evaluationcited by 24
  7. 2021Policy Learning with Adaptively Collected Datacited by 21
  8. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  9. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  10. 2022PAC-Bayesian Offline Contextual Bandits With Guaranteescited by 19
  11. 2024Off-Policy Evaluation of Slate Bandit Policies via Optimizing Abstractioncited by 17
  12. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
  13. 2023Off-Policy Evaluation of Ranking Policies under Diverse User Behaviorcited by 16
  14. 2024Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learningcited by 16
  15. 2021Off-policy Confidence Sequencescited by 15
  16. 2022Safe Exploration for Efficient Policy Evaluation and Comparisoncited by 15
  17. 2023Exponential Smoothing for Off-Policy Learningcited by 15
  18. 2021Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settingscited by 14
  19. 2023Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Banditscited by 14
  20. 2023Off-Policy Evaluation for Large Action Spaces via Policy Convolutioncited by 14
  21. 2021Control Variates for Slate Off-Policy Evaluationcited by 13
  22. 2024POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy Decompositioncited by 13
  23. 2021Bandit Algorithms for Precision Medicinecited by 12
  24. 2023A Survey of Demonstration Learningcited by 12
  25. 2022Uncertainty Quantification for Fairness in Two-Stage Recommender Systemscited by 10
  26. 2022Off-policy estimation of linear functionals: Non-asymptotic theory for semi-parametric efficiencycited by 10
  27. 2021Minimax Off-Policy Evaluation for Multi-Armed Banditscited by 9
  28. 2022Doubly robust nearest neighbors in factor modelscited by 9
  29. 2023Wasserstein Distributionally Robust Policy Evaluation and Learning for Contextual Banditscited by 9
  30. 2023Marginal Density Ratio for Off-Policy Evaluation in Contextual Banditscited by 9
  31. 2024Long-term Off-Policy Evaluation and Learningcited by 9
  32. 2021Combining Online Learning and Offline Learning for Contextual Bandits with Deficient Supportcited by 7
  33. 2022Off-Policy Risk Assessment in Markov Decision Processescited by 7
  34. 2023Learning Action Embeddings for Off-Policy Evaluationcited by 7
  35. 2023Kernel-based off-policy estimation without overlap: Instance optimality beyond semiparametric efficiencycited by 6
  36. 2023Individualized Policy Evaluation and Learning under Clustered Network Interferencecited by 6
  37. 2023Balanced Off-Policy Evaluation for Personalized Pricingcited by 5
  38. 2024Bayesian Off-Policy Evaluation and Learning for Large Action Spacescited by 5

50 most cited of 111 All 111 (JSON)