# Efficient Counterfactual Learning from Bandit Feedback

arXiv: https://arxiv.org/abs/1809.03084 · HTML: https://mlsanity.com/paper/1809.03084/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2018.

## Built on by (most cited first, 15 of 15)

- [A Review of Off-Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2212.06355/) (2022)
- [Off-Policy Evaluation for Large Action Spaces via Embeddings](https://mlsanity.com/paper/2202.06317/) (2022)
- [Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling](https://mlsanity.com/paper/2305.08062/) (2023)
- [Evaluating the Robustness of Off-Policy Evaluation](https://mlsanity.com/paper/2108.13703/) (2021)
- [Robust On-Policy Sampling for Data-Efficient Policy Evaluation in Reinforcement Learning](https://mlsanity.com/paper/2111.14552/) (2021)
- [Active Adaptive Experimental Design for Treatment Effect Estimation with Covariate Choices](https://mlsanity.com/paper/2403.03589/) (2024)
- [Top-k eXtreme Contextual Bandits with Arm Hierarchy](https://mlsanity.com/paper/2102.07800/) (2021)
- [Offline Policy Optimization with Eligible Actions](https://mlsanity.com/paper/2207.00632/) (2022)
- [On-Policy Policy Gradient Reinforcement Learning Without On-Policy Sampling](https://mlsanity.com/paper/2311.08290/) (2023)
- [Off-policy estimation with adaptively collected data: the power of online learning](https://mlsanity.com/paper/2411.12786/) (2024)
- [Bayesian Inference of Contextual Bandit Policies via Empirical Likelihood](https://mlsanity.com/paper/2602.10608/) (2026)
- [Dynamic Selection in Algorithmic Decision-making](https://mlsanity.com/paper/2108.12547/) (2021)
- [Adaptive Doubly Robust Estimator from Non-stationary Logging Policy under a Convergence of Average Probability](https://mlsanity.com/paper/2102.08975/) (2021)
- [Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning](https://mlsanity.com/paper/2308.14897/) (2023)
- [Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policies](https://mlsanity.com/paper/2508.01049/) (2025)

## Abstract (arXiv)

What is the most statistically efficient way to do off-policy evaluation and optimization with batch data from bandit feedback? For log data generated by contextual bandit algorithms, we consider offline estimators for the expected reward from a counterfactual policy. Our estimators are shown to have lowest variance in a wide class of estimators, achieving variance reduction relative to standard estimators. We then apply our estimators to improve advertisement design by a major advertisement company. Consistent with the theoretical result, our estimators allow us to improve on the existing bandit algorithm with more statistical confidence compared to a state-of-the-art benchmark.