mlsanityData to 14 Oct 2026

Off-policy estimation with adaptively collected data: the power of online learning

arXiv 2411.12786 ↗First posted 19 Nov 2024stat.ML cs.LG math.OCcited by 3

Not on the map. Why?

Abstract (arXiv)

We consider estimation of a linear functional of the treatment effect using adaptively collected data. This task finds a variety of applications including the off-policy evaluation (textsfOPE) in contextual bandits, and estimation of the average treatment effect (textsfATE) in causal inference. While a certain class of augmented inverse propensity weighting (textsfAIPW) estimators enjoys desirable asymptotic properties including the semi-parametric efficiency, much less is known about their non-asymptotic theory with adaptively collected data. To fill in the gap, we first establish generic upper bounds on the mean-squared error of the class of AIPW estimators that crucially depends on a sequentially weighted error between the treatment effect and its estimates. Motivated by this, we also propose a general reduction scheme that allows one to produce a sequence of estimates for the treatment effect via online learning to minimize the sequentially weighted estimation error. To illustrate this, we provide three concrete instantiations in (romannumeral 1) the tabular case; (romannumeral 2) the case of linear function approximation; and (romannumeral 3) the case of general function approximation for the outcome model. We then provide a local minimax lower bound to show the instance-dependent optimality of the textsfAIPW estimator using no-regret online learning algorithms.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 30 of its 77 references that are in the map. First posted on arXiv 19 Nov 2024.

Builds on

30 of 77 references.

References · 30

  1. 2019Introduction to Online Convex Optimizationcited by 803
  2. 2017A Tutorial on Thompson Samplingcited by 433
  3. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  4. 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 229
  5. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  6. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  7. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  8. 2019Confidence Intervals for Policy Evaluation in Adaptive Experimentscited by 95
  9. 2019Doubly robust off-policy evaluation with shrinkagecited by 87
  10. 2017Double/Debiased/Neyman Machine Learning of Treatment Effectscited by 76
  11. 2020Inference for Batched Banditscited by 63
  12. 2018Policy Evaluation and Optimization with Continuous Treatmentscited by 58
18 more
  1. 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 54
  2. 2021Off-Policy Evaluation via Adaptive Weighting with Data from Contextual Banditscited by 40
  3. 2022Anytime-valid off-policy inference for contextual banditscited by 37
  4. 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
  5. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  6. 2020Optimal Off-Policy Evaluation from Multiple Logging Policiescited by 30
  7. 2020Efficient Adaptive Experimental Design for Average Treatment Effect Estimationcited by 23
  8. 2023Semiparametric Efficient Inference in Adaptive Experimentscited by 23
  9. 2021Policy Learning with Adaptively Collected Datacited by 21
  10. 2017Effective Evaluation using Logged Bandit Feedback from Multiple Loggerscited by 20
  11. 2023Clip-OGD: An Experimental Design for Adaptive Neyman Allocation in Sequential Experimentscited by 16
  12. 2018Efficient Counterfactual Learning from Bandit Feedbackcited by 15
  13. 2021Off-policy Confidence Sequencescited by 15
  14. 2023Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Banditscited by 15
  15. 2017Finite-Sample Optimal Estimation and Inference on Average Treatment Effects Under Unconfoundednesscited by 11
  16. 2022Off-policy estimation of linear functionals: Non-asymptotic theory for semi-parametric efficiencycited by 10
  17. 2021Minimax Off-Policy Evaluation for Multi-Armed Banditscited by 9
  18. 2022Local Metric Learning for Off-Policy Evaluation in Contextual Bandits with Continuous Actionscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Efficient Adaptive Experimentation with Noncompliancecited by 2
  2. 2026Anytime-valid Optimal Policy Identificationcited by 1
  3. 2026Adaptive Experimentation for Censored Survival Outcomescited by 0