# Variance reduction combining pre-experiment and in-experiment data

arXiv: https://arxiv.org/abs/2410.09027 · HTML: https://mlsanity.com/paper/2410.09027/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 36 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Builds on (most cited first, 10 of 10)

- [Machine Learning for Variance Reduction in Online Experiments](https://mlsanity.com/paper/2106.07263/) (2021)
- [A First Course in Causal Inference](https://mlsanity.com/paper/2305.18793/) (2023)
- [On boosting the power of Chatterjee's rank correlation](https://mlsanity.com/paper/2108.06828/) (2021)
- [Multiply robust estimation of causal effects under principal ignorability](https://mlsanity.com/paper/2012.01615/) (2020)
- [Principal stratification analysis using principal scores](https://mlsanity.com/paper/1602.01196/) (2016)
- [Towards Optimal Variance Reduction in Online Controlled Experiments](https://mlsanity.com/paper/2110.13406/) (2021)
- [No-harm calibration for generalized Oaxaca-Blinder estimators](https://mlsanity.com/paper/2012.09246/) (2020)
- [Limit theorems of Chatterjee's rank correlation](https://mlsanity.com/paper/2204.08031/) (2022)
- [Zero to Hero: Exploiting Null Effects to Achieve Variance Reduction in Experiments with One-sided Triggering](https://mlsanity.com/paper/2112.13299/) (2021)
- [Forward selection and post-selection inference in factorial designs](https://mlsanity.com/paper/2301.12045/) (2023)

## Built on by (most cited first, 2 of 2)

- [Exploiting Similarities in A/B Testing with Off-Policy Estimation](https://mlsanity.com/paper/2506.10677/) (2025)
- [A More Accurate Algorithm Comparison through A/B Testing using Offline Evaluation Methods](https://mlsanity.com/paper/2607.01958/) (2026)

## Abstract (arXiv)

Online controlled experiments (A/B testing) are fundamental to data-driven decision-making in many companies. Improving the sensitivity of these experiments under fixed sample size constraints requires reducing the variance of the average treatment effect (ATE) estimator. Existing variance reduction techniques such as CUPED and CUPAC use pre-experiment data, but their effectiveness depends on how predictive those data are for outcomes measured during the experiment. In-experiment data are often more strongly correlated with the outcome, but using arbitrary post-treatment variables can introduce bias. In this paper, we propose a general, robust, and scalable framework that combines both pre-experiment and in-experiment data to achieve variance reduction. Our framework is simple, interpretable, and computationally efficient, making it practical for real-world deployment. We develop the asymptotic theory of the proposed estimator and provide consistent variance estimators. Empirical results from multiple online experiments conducted at Etsy demonstrate substantial additional variance reduction over current pipeline, even when incorporating only a few post-treatment covariates. These findings underscore the effectiveness of our framework in improving experimental sensitivity and accelerating data-driven decision-making.