# Machine Learning for Variance Reduction in Online Experiments

arXiv: https://arxiv.org/abs/2106.07263 · HTML: https://mlsanity.com/paper/2106.07263/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 50 references that are in the map.
- First posted on arXiv 14 Jun 2021.

## Builds on (most cited first, 6 of 6)

- [Quasi-Oracle Estimation of Heterogeneous Treatment Effects](https://mlsanity.com/paper/1712.04912/) (2017)
- [Generalized Random Forests](https://mlsanity.com/paper/1610.01271/) (2016)
- [High-dimensional regression adjustments in randomized experiments](https://mlsanity.com/paper/1607.06801/) (2016)
- [Optimization by gradient boosting](https://mlsanity.com/paper/1707.05023/) (2017)
- [No-harm calibration for generalized Oaxaca-Blinder estimators](https://mlsanity.com/paper/2012.09246/) (2020)
- [The Generalized Oaxaca-Blinder Estimator](https://mlsanity.com/paper/2004.11615/) (2020)

## Built on by (most cited first, 17 of 17)

- [Variance Reduction in Ratio Metrics for Efficient Online Experiments](https://mlsanity.com/paper/2401.04062/) (2024)
- [Learning Metrics that Maximise Power for Accelerated A/B-Tests](https://mlsanity.com/paper/2402.03915/) (2024)
- [On Efficient Estimation of Distributional Treatment Effects under Covariate-Adaptive Randomization](https://mlsanity.com/paper/2506.05945/) (2025)
- [Estimating Distributional Treatment Effects in Randomized Experiments: Machine Learning for Variance Reduction](https://mlsanity.com/paper/2407.16037/) (2024)
- [t-Testing the Waters: Empirically Validating Assumptions for Reliable A/B-Testing](https://mlsanity.com/paper/2502.04793/) (2025)
- [Variance reduction combining pre-experiment and in-experiment data](https://mlsanity.com/paper/2410.09027/) (2024)
- [Causal inference with Machine Learning-Based Covariate Representation](https://mlsanity.com/paper/2311.01709/) (2023)
- [Strategic A/B testing via Maximum Probability-driven Two-armed Bandit](https://mlsanity.com/paper/2506.22536/) (2025)
- [Subset Selection for Stratified Sampling in Online Controlled Experiments](https://mlsanity.com/paper/2509.15576/) (2025)
- [Unifying On- and Off-Policy Variance Reduction Methods](https://mlsanity.com/paper/2603.08370/) (2026)
- [A More Accurate Algorithm Comparison through A/B Testing using Offline Evaluation Methods](https://mlsanity.com/paper/2607.01958/) (2026)
- [Improving uplift model evaluation on RCT data](https://mlsanity.com/paper/2210.02152/) (2022)
- [STATE: A Robust ATE Estimator of Heavy-Tailed Metrics for Variance Reduction in Online Controlled Experiments](https://mlsanity.com/paper/2407.16337/) (2024)
- [Sustained Impact of Agentic Personalisation in Marketing: A Longitudinal Case Study](https://mlsanity.com/paper/2604.08621/) (2026)
- [Modeling Covariate Transition for Efficient Estimation of Longitudinal Treatment Effects in Randomized Experiments](https://mlsanity.com/paper/2605.31443/) (2026)
- [Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap](https://mlsanity.com/paper/2607.14604/) (2026)
- [Optimal Multi-way Decision Trees for Stratified Sampling in Online Controlled Experiments](https://mlsanity.com/paper/2609.23308/) (2026)

## Abstract (arXiv)

We consider the problem of variance reduction in randomized controlled trials, through the use of covariates correlated with the outcome but independent of the treatment. We propose a machine learning regression-adjusted treatment effect estimator, which we call MLRATE. MLRATE uses machine learning predictors of the outcome to reduce estimator variance. It employs cross-fitting to avoid overfitting biases, and we prove consistency and asymptotic normality under general conditions. MLRATE is robust to poor predictions from the machine learning step: if the predictions are uncorrelated with the outcomes, the estimator performs asymptotically no worse than the standard difference-in-means estimator, while if predictions are highly correlated with outcomes, the efficiency gains are large. In A/A tests, for a set of 48 outcome metrics commonly monitored in Facebook experiments the estimator has over 70% lower variance than the simple difference-in-means estimator, and about 19% lower variance than the common univariate procedure which adjusts only for pre-experiment values of the outcome.