mlsanityData to 7 Oct 2026

There are natural scores: Full comment on Shafer, "Testing by betting: A strategy for statistical and scientific communication"

arXiv 2102.05569 ↗First posted 10 Feb 2021math.ST stat.ME stat.THcited by 49

Not on the map. Why?

Abstract (arXiv)

Shafer (2021) offers a betting perspective on statistical testing which may be useful for foundational debates, given that disputes over such testing continue to be intense. To be helpful for researchers, however, this perspective will need more elaboration using real examples in which (a) the betting score has a justification and interpretation in terms of study goals that distinguishes it from the uncountable mathematical possibilities, and (b) the assumptions in the sampling model are uncertain. On justification, Shafer says 'No one has made a convincing case for any particular choice' of a score derived from a P-value and then states that 'the choice is fundamentally arbitrary'. Yet some (but not most) scores can be motivated by study goals (e.g., information measurement; decision making). The one I have seen repeatedly in information statistics and data mining is the surprisal, logworth or S-value s = -log(p), where the log base determines the scale. The present comment explains the rationale for this choice.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Feb 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 4; 2023: 6; 2024: 9; 2025: 12; 2026: 14.

49 papers.

Sort:

  1. 2021Tight Concentrations and Confidence Sequences from the Regret of Universal Portfoliocited by 42
  2. 2022Anytime-valid off-policy inference for contextual banditscited by 37
  3. 2021Time-uniform central limit theory and asymptotic confidence sequencescited by 29
  4. 2023Auditing Fairness by Bettingcited by 24
  5. 2023Sequential Predictive Two-Sample and Independence Testingcited by 20
  6. 2022Sequential Kernelized Independence Testingcited by 15
  7. 2022Model-X Sequential Testing for Conditional Independence via Testing by Bettingcited by 14
  8. 2024Active, anytime-valid risk controlling prediction setscited by 14
  9. 2024Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selectioncited by 13
  10. 2024Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Bettingcited by 11
  11. 2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guaranteescited by 9
  12. 2021A unified framework for bandit multiple testingcited by 8
37 more
  1. 2021Comparing Sequential Forecasterscited by 8
  2. 2024An Auditing Test To Detect Behavioral Shift in Language Modelscited by 8
  3. 2026PACE: Anytime-Valid Acceptance Tests for Self-Evolving Agentscited by 8
  4. 2023Deep anytime-valid hypothesis testingcited by 7
  5. 2022Nonparametric extensions of randomized response for private confidence setscited by 6
  6. 2023Risk-limiting Financial Audits via Weighted Sampling without Replacementcited by 6
  7. 2025Sequentially Auditing Differential Privacycited by 5
  8. 2025Theoretical guarantees for change localization using conformal p-valuescited by 5
  9. 2023Incentive-Theoretic Bayesian Inference for Collaborative Sciencecited by 4
  10. 2025Optimistic Interior Point Methods for Sequential Hypothesis Testing by Bettingcited by 4
  11. 2023The extended Ville's inequality for nonintegrable nonnegative supermartingalescited by 3
  12. 2024I Bet You Did Not Mean That: Testing Semantic Importance via Bettingcited by 3
  13. 2024Asymptotic Time-Uniform Inference for Parameters in Averaged Stochastic Approximationcited by 3
  14. 2025From Individual Experience to Collective Evidence: A Reporting-Based Framework for Identifying Systemic Harmscited by 3
  15. 2026Testing For Distribution Shifts with Conditional Conformal Test Martingalescited by 3
  16. 2024Combining Evidence Across Filtrationscited by 2
  17. 2025Fuzzy Prediction Sets: Conformal Prediction with E-valuescited by 2
  18. 2025Eventually LIL Regret: Almost Sure lnln T Regret for a sub-Gaussian Mixture on Unbounded Datacited by 2
  19. 2026Adaptive auditing of AI systems with anytime-valid guaranteescited by 2
  20. 2024Sequential Kernelized Stein Discrepancycited by 1
  21. 2025Extending Prediction-Powered Inference through Conformal Predictioncited by 1
  22. 2025E-Scores for (In)Correctness Assessment of Generative Model Outputscited by 1
  23. 2026Global Sequential Testing for Multi-Stream Auditingcited by 1
  24. 2026Cross-Domain Uncertainty Quantification for Selective Prediction: A Comprehensive Bound Ablation with Transfer-Informed Bettingcited by 1
  25. 2026Multi-Armed Sequential Hypothesis Testing by Bettingcited by 1
  26. 2026Optimal Rates for Differentially Private Hypothesis Testing with E-valuescited by 1
  27. 2026Anytime-Valid Confirmation of Label-Shift Correctionscited by 1
  28. 2024Data-light Uncertainty Set Merging with Admissibilitycited by 0
  29. 2025On the Precise Asymptotics of Universal Inferencecited by 0
  30. 2025Matched-Pair Experimental Design with Active Learningcited by 0
  31. 2025Assessing Extrapolation of Peaks Over Thresholds with Martingale Testingcited by 0
  32. 2026Learning to Bet for Horizon-Aware Anytime-Valid Testingcited by 0
  33. 2026Spectral Structure in Finite Free Information Inequalities and p-Stam Phase Transitionscited by 0
  34. 2026Recipes for Calibration Checks in Safety-Critical Applicationscited by 0
  35. 2026Optimal sequential tests yield log-optimal e-processescited by 0
  36. 2026Semi-Supervised Hypothesis Testing by Betting on Predictionscited by 0
  37. 2026NxN E-valuation: Hypothesis Certification via a Conformal CRT Nullcited by 0