Exploiting Similarities in A/B Testing with Off-Policy Estimation
Not on the map. Why?
Abstract (arXiv)
We study A/B testing, the standard protocol for measuring the performance gain of a new decision system relative to a baseline. Traditional A/B testing treats both systems as black boxes, ignoring potential similarities between them. In practice, however, new and baseline systems are rarely radically different and often share significant structure, which can be captured by their propensities to make similar decisions. We show that in such cases, the commonly used difference-in-means estimator, though unbiased, is statistically suboptimal. Leveraging off-policy estimation, we introduce a family of A/B testing estimators that exploit the propensities of the tested systems to achieve improved concentration properties. This family is flexible enough to be tailored to practical decision-making. The resulting estimators are simple, robust to propensities misspecification, substantially more accurate when the tested systems exhibit similarities, and gracefully fall back to the difference-in-means estimator when such similarities are absent. Our theoretical analysis and empirical studies confirm their efficiency and practicality.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 18 of its 40 references that are in the map. First posted on arXiv 12 Jun 2025.
Builds on
18 of 40 references.
References · 18
- 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
- 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
- 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
- 2018Offline A/B testing for Recommender Systemscited by 96
- 2019Doubly robust off-policy evaluation with shrinkagecited by 87
- 2020Off-policy Bandits with Deficient Supportcited by 61
- 2022Off-Policy Evaluation for Large Action Spaces via Embeddingscited by 54
- 2019Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Samplingcited by 37
- 2020Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Frameworkcited by 29
- 2017Effective Evaluation using Logged Bandit Feedback from Multiple Loggerscited by 20
- 2022PAC-Bayesian Offline Contextual Bandits With Guaranteescited by 20
- 2020BLOB : A Probabilistic Model for Recommendation that Combines Organic and Bandit Signalscited by 19
6 more
- 2024Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learningcited by 16
- 2023Exponential Smoothing for Off-Policy Learningcited by 15
- 2023Off-Policy Evaluation for Large Action Spaces via Policy Convolutioncited by 14
- 2019Conditional Importance Sampling for Off-Policy Learningcited by 13
- 2023Correcting for Interference in Experiments: A Case Study at Douyincited by 12
- 2024Variance reduction combining pre-experiment and in-experiment datacited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.