# Variance Reduction in Ratio Metrics for Efficient Online Experiments

arXiv: https://arxiv.org/abs/2401.04062 · HTML: https://mlsanity.com/paper/2401.04062/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 11 references that are in the map.
- First posted on arXiv 8 Jan 2024.

## Builds on (most cited first, 2 of 2)

- [Machine Learning for Variance Reduction in Online Experiments](https://mlsanity.com/paper/2106.07263/) (2021)
- [A Common Misassumption in Online Experiments with Machine Learning Models](https://mlsanity.com/paper/2304.10900/) (2023)

## Built on by (most cited first, 6 of 6)

- [Optimal Baseline Corrections for Off-Policy Contextual Bandits](https://mlsanity.com/paper/2405.05736/) (2024)
- [Multi-Objective Recommendation via Multivariate Policy Learning](https://mlsanity.com/paper/2405.02141/) (2024)
- [Learning Metrics that Maximise Power for Accelerated A/B-Tests](https://mlsanity.com/paper/2402.03915/) (2024)
- [t-Testing the Waters: Empirically Validating Assumptions for Reliable A/B-Testing](https://mlsanity.com/paper/2502.04793/) (2025)
- [Unifying On- and Off-Policy Variance Reduction Methods](https://mlsanity.com/paper/2603.08370/) (2026)
- [Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap](https://mlsanity.com/paper/2607.14604/) (2026)

## Abstract (arXiv)

Online controlled experiments, such as A/B-tests, are commonly used by modern tech companies to enable continuous system improvements. Despite their paramount importance, A/B-tests are expensive: by their very definition, a percentage of traffic is assigned an inferior system variant. To ensure statistical significance on top-level metrics, online experiments typically run for several weeks. Even then, a considerable amount of experiments will lead to inconclusive results (i.e. false negatives, or type-II error). The main culprit for this inefficiency is the variance of the online metrics. Variance reduction techniques have been proposed in the literature, but their direct applicability to commonly used ratio metrics (e.g. click-through rate or user retention) is limited. In this work, we successfully apply variance reduction techniques to ratio metrics on a large-scale short-video platform: ShareChat. Our empirical results show that we can either improve A/B-test confidence in 77% of cases, or can retain the same level of confidence with 30% fewer data points. Importantly, we show that the common approach of including as many covariates as possible in regression is counter-productive, highlighting that control variates based on Gradient-Boosted Decision Tree predictors are most effective. We discuss the practicalities of implementing these methods at scale and showcase the cost reduction they beget.