# Post-hoc α Hypothesis Testing and the Post-hoc p-value

arXiv: https://arxiv.org/abs/2312.08040 · HTML: https://mlsanity.com/paper/2312.08040/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2023.

## Built on by (most cited first, 3 of 3)

- [Anytime-valid simulation-based hypothesis testing](https://mlsanity.com/paper/2610.08210/) (2026)
- [Set-Preserving Calibration from Conformal P-Values to E-Values](https://mlsanity.com/paper/2606.03600/) (2026)
- [Possibilistic Radial Transport for Approximate IM Inference](https://mlsanity.com/paper/2610.09956/) (2026)

## Abstract (arXiv)

In traditional hypothesis testing one must pre-specify the significance level α to bound the `size' of the test: its probability to falsely reject the hypothesis. Indeed, a data-dependent selection of α would generally distort the size, possibly making it larger than the specified level α. We explore hypothesis testing with a data-dependent choice of α by guaranteeing that there is no such size distortion in expectation, even if the level α is arbitrarily selected based on the data. Unlike regular p-values, resulting `post-hoc p-values' allow us to `reject at level p' and still provide this guarantee. Interestingly, we find that p is a post-hoc p-value if and only if 1/p is an e-value, a recently introduced measure of evidence. While often treated as different paradigms, this reveals e-values are simply p-values under a stronger error guarantee, thinly veiled by the reciprocal p = 1/e. Moreover, we extend classical optimal testing to optimal post-hoc testing. Finally, we apply our work to close Markov's inequality into a post-hoc α equality, and we study more general forms of post-hoc testing that require us to generalize beyond e-values.