# On admissibility in post-hoc hypothesis testing

arXiv: https://arxiv.org/abs/2508.00770 · HTML: https://mlsanity.com/paper/2508.00770/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Aug 2025.

## Built on by (most cited first, 3 of 3)

- [Adaptive Coverage Policies in Conformal Prediction](https://mlsanity.com/paper/2510.04318/) (2025)
- [Incentive Aware AI Regulations: A Credal Characterisation](https://mlsanity.com/paper/2603.05175/) (2026)
- [Bentkus-type asymptotic e-values](https://mlsanity.com/paper/2606.06332/) (2026)

## Abstract (arXiv)

The validity of classical hypothesis testing requires the significance level α be fixed before any statistical analysis takes place. This is a stringent requirement. For instance, it prohibits updating α during (or after) an experiment due to changing concern about the cost of false positives, or to reflect unexpectedly strong evidence against the null. Perhaps most disturbingly, witnessing a p-value pllα vs p= α- ε for tiny ε > 0 has no (statistical) relevance for any downstream decision-making. Following recent work of Gr"unwald (2024), we develop a theory of post-hoc hypothesis testing, enabling α to be chosen after seeing and analyzing the data. To study "good" post-hoc tests we introduce Gamma-admissibility, where Gamma is a set of adversaries which map the data to a significance level. We classify the set of Gamma-admissible rules for various sets Gamma, showing they must be based on e-values, and recover the Neyman-Pearson lemma when Gamma is the constant map.