# A Bandit Approach to Multiple Testing with False Discovery Control

arXiv: https://arxiv.org/abs/1809.02235 · HTML: https://mlsanity.com/paper/1809.02235/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Sep 2018.

## Built on by (most cited first, 7 of 7)

- [Sequential Harmful Shift Detection Without Labels](https://mlsanity.com/paper/2412.12910/) (2024)
- [Comparing Sequential Forecasters](https://mlsanity.com/paper/2110.00115/) (2021)
- [Challenges in Statistical Analysis of Data Collected by a Bandit Algorithm: An Empirical Exploration in Applications to Adaptively Randomized Experiments](https://mlsanity.com/paper/2103.12198/) (2021)
- [Online Sign Identification: Minimization of the Number of Errors in Thresholding Bandits](https://mlsanity.com/paper/2110.09133/) (2021)
- [Efficient Determination of Safety Requirements for Perception Systems](https://mlsanity.com/paper/2307.01371/) (2023)
- [Nearest Neighbor Search Under Uncertainty](https://mlsanity.com/paper/2103.05057/) (2021)
- [Practical, Provably-Correct Interactive Learning in the Realizable Setting: The Power of True Believers](https://mlsanity.com/paper/2111.04915/) (2021)

## Abstract (arXiv)

We propose an adaptive sampling approach for multiple testing which aims to maximize statistical power while ensuring anytime false discovery control. We consider n distributions whose means are partitioned by whether they are below or equal to a baseline (nulls), versus above the baseline (actual positives). In addition, each distribution can be sequentially and repeatedly sampled. Inspired by the multi-armed bandit literature, we provide an algorithm that takes as few samples as possible to exceed a target true positive proportion (i.e. proportion of actual positives discovered) while giving anytime control of the false discovery proportion (nulls predicted as actual positives). Our sample complexity results match known information theoretic lower bounds and through simulations we show a substantial performance improvement over uniform sampling and an adaptive elimination style algorithm. Given the simplicity of the approach, and its sample efficiency, the method has promise for wide adoption in the biological sciences, clinical testing for drug discovery, and online A/B/n testing problems.