# Valid sequential inference on probability forecast performance

arXiv: https://arxiv.org/abs/2103.08402 · HTML: https://mlsanity.com/paper/2103.08402/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2021.

## Built on by (most cited first, 7 of 7)

- [Proper scoring rules for estimation and forecast evaluation](https://mlsanity.com/paper/2504.01781/) (2025)
- [Model Comparison and Calibration Assessment: User Guide for Consistent Scoring Functions in Machine Learning and Actuarial Practice](https://mlsanity.com/paper/2202.12780/) (2022)
- [Comparing Sequential Forecasters](https://mlsanity.com/paper/2110.00115/) (2021)
- [Combining Evidence Across Filtrations](https://mlsanity.com/paper/2402.09698/) (2024)
- [Generic E-Variables for Exact Sequential k-Sample Tests that allow for Optional Stopping](https://mlsanity.com/paper/2106.02693/) (2021)
- [Sequential Structure-Sensitive Residual Diagnostics for PDE Inverse Problems](https://mlsanity.com/paper/2607.02101/) (2026)
- [Assessing Extrapolation of Peaks Over Thresholds with Martingale Testing](https://mlsanity.com/paper/2512.03116/) (2025)

## Abstract (arXiv)

Probability forecasts for binary events play a central role in many applications. Their quality is commonly assessed with proper scoring rules, which assign forecasts a numerical score such that a correct forecast achieves a minimal expected score. In this paper, we construct e-values for testing the statistical significance of score differences of competing forecasts in sequential settings. E-values have been proposed as an alternative to p-values for hypothesis testing, and they can easily be transformed into conservative p-values by taking the multiplicative inverse. The e-values proposed in this article are valid in finite samples without any assumptions on the data generating processes. They also allow optional stopping, so a forecast user may decide to interrupt evaluation taking into account the available data at any time and still draw statistically valid inference, which is generally not true for classical p-value based tests. In a case study on postprocessing of precipitation forecasts, state-of-the-art forecasts dominance tests and e-values lead to the same conclusions.