# Large-scale Validation of Counterfactual Learning Methods: A Test-Bed

arXiv: https://arxiv.org/abs/1612.00367 · HTML: https://mlsanity.com/paper/1612.00367/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2016.

## Built on by (most cited first, 8 of 8)

- [On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models](https://mlsanity.com/paper/2209.05310/) (2022)
- [A Large Scale Benchmark for Individual Treatment Effect Prediction and Uplift Modeling](https://mlsanity.com/paper/2111.10106/) (2021)
- [Enhancing Counterfactual Classification via Self-Training](https://mlsanity.com/paper/2112.04461/) (2021)
- [Synthetic Data-Based Simulators for Recommender Systems: A Survey](https://mlsanity.com/paper/2206.11338/) (2022)
- [A Field Test of Bandit Algorithms for Recommendations: Understanding the Validity of Assumptions on Human Preferences in Multi-armed Bandits](https://mlsanity.com/paper/2304.09088/) (2023)
- [AExGym: Benchmarks and Environments for Adaptive Experimentation](https://mlsanity.com/paper/2408.04531/) (2024)
- [Meta Off-Policy Estimation](https://mlsanity.com/paper/2508.07914/) (2025)
- [Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap](https://mlsanity.com/paper/2607.14604/) (2026)

## Abstract (arXiv)

The ability to perform effective off-policy learning would revolutionize the process of building better interactive systems, such as search engines and recommendation systems for e-commerce, computational advertising and news. Recent approaches for off-policy evaluation and learning in these settings appear promising. With this paper, we provide real-world data and a standardized test-bed to systematically investigate these algorithms using data from display advertising. In particular, we consider the problem of filling a banner ad with an aggregate of multiple products the user may want to purchase. This paper presents our test-bed, the sanity checks we ran to ensure its validity, and shows results comparing state-of-the-art off-policy learning methods like doubly robust optimization, POEM, and reductions to supervised learning using regression baselines. Our results show experimental evidence that recent off-policy learning methods can improve upon state-of-the-art supervised learning techniques on a large-scale real-world data set.