# Empirical Bayes Multistage Testing for Large-Scale Experiments

arXiv: https://arxiv.org/abs/2209.05788 · HTML: https://mlsanity.com/paper/2209.05788/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2022.

## Built on by (most cited first, 1 of 1)

- [Rapid and Scalable Bayesian AB Testing](https://mlsanity.com/paper/2307.14628/) (2023)

## Abstract (arXiv)

Modern application of A/B tests is challenging due to its large scale in various dimensions, which demands flexibility to deal with multiple testing sequentially. The state-of-the-art practice first reduces the observed data stream to always-valid p-values, and then chooses a cut-off as in conventional multiple testing schemes. Here we propose an alternative method called AMSET (adaptive multistage empirical Bayes test) by incorporating historical data in decision-making to achieve efficiency gains while retaining marginal false discovery rate (mFDR) control that is immune to peeking. We also show that a fully data-driven estimation in AMSET performs robustly to various simulation and real data settings at a large mobile app social network company.