Efficient Counterfactual Learning from Bandit Feedback
Not on the map. Why?
Abstract (arXiv)
What is the most statistically efficient way to do off-policy evaluation and optimization with batch data from bandit feedback? For log data generated by contextual bandit algorithms, we consider offline estimators for the expected reward from a counterfactual policy. Our estimators are shown to have lowest variance in a wide class of estimators, achieving variance reduction relative to standard estimators. We then apply our estimators to improve advertisement design by a major advertisement company. Consistent with the theoretical result, our estimators allow us to improve on the existing bandit algorithm with more statistical confidence compared to a state-of-the-art benchmark.
Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Sep 2018.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 5; 2022: 3; 2023: 3; 2024: 2; 2025: 1; 2026: 1.
15 papers.
Sort:
By area:
- 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
- 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 53
- 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
- 2021Evaluating the Robustness of Off-Policy Evaluationcited by 32
- 2021Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learningcited by 15
- 2024Active Adaptive Experimental Design for Treatment Effect Estimation with Covariate Choicescited by 12
- 2021Top-k eXtreme Contextual Bandits with Arm Hierarchycited by 8
- 2022Offline Policy Optimization with Eligible Actionscited by 7
- 2023On-Policy Policy Gradient Reinforcement Learning Without On-Policy Samplingcited by 6
- 2024Off-policy estimation with adaptively collected data: the power of online learningcited by 3
- 2021Dynamic Selection in Algorithmic Decision-makingcited by 1
- 2026Bayesian Inference of Contextual Bandit Policies via Empirical Likelihoodcited by 1
3 more
- 2021Adaptive Doubly Robust Estimator from Non-stationary Logging Policy under a Convergence of Average Probabilitycited by 0
- 2023Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learningcited by 0
- 2025Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policiescited by 0
- 2026Bayesian Inference of Contextual Bandit Policies via Empirical Likelihood11 Feb 2026
- 2025Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policies1 Aug 2025
- 2024Off-policy estimation with adaptively collected data: the power of online learning19 Nov 2024
- 2024Active Adaptive Experimental Design for Treatment Effect Estimation with Covariate Choices6 Mar 2024
- 2023On-Policy Policy Gradient Reinforcement Learning Without On-Policy Sampling14 Nov 2023
- 2023Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning28 Aug 2023
- 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling14 May 2023
- 2022A Review of Off-Policy Evaluation in Reinforcement Learning13 Dec 2022
- 2022Offline Policy Optimization with Eligible Actions1 Jul 2022
- 2022Off-Policy Evaluation for Large Action Spaces via Embeddings13 Feb 2022
- 2021Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learning29 Nov 2021
- 2021Evaluating the Robustness of Off-Policy Evaluation31 Aug 2021
- 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
- 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 53
- 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modelingcited by 34
- 2021Evaluating the Robustness of Off-Policy Evaluationcited by 32
- 2021Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learningcited by 15
- 2024Active Adaptive Experimental Design for Treatment Effect Estimation with Covariate Choicescited by 12
- 2021Top-k eXtreme Contextual Bandits with Arm Hierarchycited by 8
- 2023On-Policy Policy Gradient Reinforcement Learning Without On-Policy Samplingcited by 6
- 2024Off-policy estimation with adaptively collected data: the power of online learningcited by 3
- 2021Dynamic Selection in Algorithmic Decision-makingcited by 1
- 2026Bayesian Inference of Contextual Bandit Policies via Empirical Likelihoodcited by 1
- 2021Adaptive Doubly Robust Estimator from Non-stationary Logging Policy under a Convergence of Average Probabilitycited by 0
12 most cited of 13 All 13 (JSON)
- 2026Bayesian Inference of Contextual Bandit Policies via Empirical Likelihood11 Feb 2026
- 2025Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policies1 Aug 2025
- 2024Off-policy estimation with adaptively collected data: the power of online learning19 Nov 2024
- 2024Active Adaptive Experimental Design for Treatment Effect Estimation with Covariate Choices6 Mar 2024
- 2023On-Policy Policy Gradient Reinforcement Learning Without On-Policy Sampling14 Nov 2023
- 2023Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling14 May 2023
- 2022A Review of Off-Policy Evaluation in Reinforcement Learning13 Dec 2022
- 2022Off-Policy Evaluation for Large Action Spaces via Embeddings13 Feb 2022
- 2021Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learning29 Nov 2021
- 2021Evaluating the Robustness of Off-Policy Evaluation31 Aug 2021
- 2021Dynamic Selection in Algorithmic Decision-making28 Aug 2021
- 2021Adaptive Doubly Robust Estimator from Non-stationary Logging Policy under a Convergence of Average Probability17 Feb 2021
12 newest of 13 All 13 (JSON)