# Multiple testing with anytime-valid Monte Carlo p-values

arXiv: https://arxiv.org/abs/2404.15586 · HTML: https://mlsanity.com/paper/2404.15586/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2024.

## Built on by (most cited first, 2 of 2)

- [Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2410.22318/) (2024)
- [Optimistic Interior Point Methods for Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2502.07774/) (2025)

## Abstract (arXiv)

In contemporary problems involving genetic or neuroimaging data, thousands of hypotheses need to be tested. Due to their high power, and finite sample guarantees on type-I error under weak assumptions, Monte Carlo permutation tests are often considered as gold standard for these settings. However, the enormous computational effort required for (thousands of) permutation tests is a major burden. In this paper, we integrate recently constructed anytime-valid permutation p-values into a broad class of multiple testing procedures, including the Benjamini-Hochberg procedure. This allows to fully adapt the number of permutations to the underlying data and thus, for example, to the number of rejections made by the multiple testing procedure. Even though this data-adaptive stopping can induce dependencies between the p-values that violate the usual assumptions of the Benjamini-Hochberg procedure, we prove that our approach controls the false discovery rate under mild assumptions. Furthermore, our method provably decreases the required number of permutations substantially without compromising power. On a real genomics data set, our method reduced the computational time from more than three days to less than four minutes while increasing the number of rejections.