# Percent Change Estimation in Large Scale Online Experiments

arXiv: https://arxiv.org/abs/1711.00562 · HTML: https://mlsanity.com/paper/1711.00562/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2017.

## Built on by (most cited first, 2 of 2)

- [Mondegreen: A Post-Processing Solution to Speech Recognition Error Correction for Voice Search Queries](https://mlsanity.com/paper/2105.09930/) (2021)
- [Setting the duration of online A/B experiments](https://mlsanity.com/paper/2408.02830/) (2024)

## Abstract (arXiv)

Online experiments are a fundamental component of the development of web-facing products. Given their large user-bases, even small product improvements can have a large impact on user engagement or profits on an absolute scale. As a result, accurately estimating the relative impact of these changes is extremely important. I propose an approach based on an objective Bayesian model to improve the sensitivity of percent change estimation in A/B experiments. Leveraging pre-period information, this approach produces more robust and accurate point estimates and up to 50% tighter credible intervals than traditional methods. The R package abpackage provides an implementation of the approach.