# There are natural scores: Full comment on Shafer, "Testing by betting: A strategy for statistical and scientific communication"

arXiv: https://arxiv.org/abs/2102.05569 · HTML: https://mlsanity.com/paper/2102.05569/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2021.

## Built on by (most cited first, 30 of 53)

- [Tight Concentrations and Confidence Sequences from the Regret of Universal Portfolio](https://mlsanity.com/paper/2110.14099/) (2021)
- [Anytime-valid off-policy inference for contextual bandits](https://mlsanity.com/paper/2210.10768/) (2022)
- [Time-uniform central limit theory and asymptotic confidence sequences](https://mlsanity.com/paper/2103.06476/) (2021)
- [Auditing Fairness by Betting](https://mlsanity.com/paper/2305.17570/) (2023)
- [Sequential Predictive Two-Sample and Independence Testing](https://mlsanity.com/paper/2305.00143/) (2023)
- [Sequential Kernelized Independence Testing](https://mlsanity.com/paper/2212.07383/) (2022)
- [Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection](https://mlsanity.com/paper/2409.15844/) (2024)
- [Model-X Sequential Testing for Conditional Independence via Testing by Betting](https://mlsanity.com/paper/2210.00354/) (2022)
- [Active, anytime-valid risk controlling prediction sets](https://mlsanity.com/paper/2406.10490/) (2024)
- [PACE: Anytime-Valid Acceptance Tests for Self-Evolving Agents](https://mlsanity.com/paper/2606.08106/) (2026)
- [Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2410.22318/) (2024)
- [Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees](https://mlsanity.com/paper/2505.18659/) (2025)
- [An Auditing Test To Detect Behavioral Shift in Language Models](https://mlsanity.com/paper/2410.19406/) (2024)
- [A unified framework for bandit multiple testing](https://mlsanity.com/paper/2107.07322/) (2021)
- [Comparing Sequential Forecasters](https://mlsanity.com/paper/2110.00115/) (2021)
- [Deep anytime-valid hypothesis testing](https://mlsanity.com/paper/2310.19384/) (2023)
- [Nonparametric extensions of randomized response for private confidence sets](https://mlsanity.com/paper/2202.08728/) (2022)
- [Risk-limiting Financial Audits via Weighted Sampling without Replacement](https://mlsanity.com/paper/2305.06884/) (2023)
- [Incentive-Theoretic Bayesian Inference for Collaborative Science](https://mlsanity.com/paper/2307.03748/) (2023)
- [Sequentially Auditing Differential Privacy](https://mlsanity.com/paper/2509.07055/) (2025)
- [Theoretical guarantees for change localization using conformal p-values](https://mlsanity.com/paper/2510.08749/) (2025)
- [Optimistic Interior Point Methods for Sequential Hypothesis Testing by Betting](https://mlsanity.com/paper/2502.07774/) (2025)
- [Testing For Distribution Shifts with Conditional Conformal Test Martingales](https://mlsanity.com/paper/2602.13848/) (2026)
- [The extended Ville's inequality for nonintegrable nonnegative supermartingales](https://mlsanity.com/paper/2304.01163/) (2023)
- [Combining Evidence Across Filtrations](https://mlsanity.com/paper/2402.09698/) (2024)
- [I Bet You Did Not Mean That: Testing Semantic Importance via Betting](https://mlsanity.com/paper/2405.19146/) (2024)
- [Asymptotic Time-Uniform Inference for Parameters in Averaged Stochastic Approximation](https://mlsanity.com/paper/2410.15057/) (2024)
- [From Individual Experience to Collective Evidence: A Reporting-Based Framework for Identifying Systemic Harms](https://mlsanity.com/paper/2502.08166/) (2025)
- [Fuzzy Prediction Sets: Conformal Prediction with E-values](https://mlsanity.com/paper/2509.13130/) (2025)
- [Adaptive auditing of AI systems with anytime-valid guarantees](https://mlsanity.com/paper/2605.07002/) (2026)

## Abstract (arXiv)

Shafer (2021) offers a betting perspective on statistical testing which may be useful for foundational debates, given that disputes over such testing continue to be intense. To be helpful for researchers, however, this perspective will need more elaboration using real examples in which (a) the betting score has a justification and interpretation in terms of study goals that distinguishes it from the uncountable mathematical possibilities, and (b) the assumptions in the sampling model are uncertain. On justification, Shafer says 'No one has made a convincing case for any particular choice' of a score derived from a P-value and then states that 'the choice is fundamentally arbitrary'. Yet some (but not most) scores can be motivated by study goals (e.g., information measurement; decision making). The one I have seen repeatedly in information statistics and data mining is the surprisal, logworth or S-value s = -log(p), where the log base determines the scale. The present comment explains the rationale for this choice.