# Local scale invariance and robustness of proper scoring rules

arXiv: https://arxiv.org/abs/1912.05642 · HTML: https://mlsanity.com/paper/1912.05642/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2019.

## Built on by (most cited first, 8 of 8)

- [Beyond Confidence: Reliable Models Should Also Consider Atypicality](https://mlsanity.com/paper/2305.18262/) (2023)
- [Proper scoring rules for estimation and forecast evaluation](https://mlsanity.com/paper/2504.01781/) (2025)
- [Probabilistic Hierarchical Forecasting with Deep Poisson Mixtures](https://mlsanity.com/paper/2110.13179/) (2021)
- [HierarchicalForecast: A Reference Framework for Hierarchical Forecasting in Python](https://mlsanity.com/paper/2207.03517/) (2022)
- [Superior Scoring Rules for Probabilistic Evaluation of Single-Label Multi-Class Classification Tasks](https://mlsanity.com/paper/2407.17697/) (2024)
- [Model selection with proper scoring rules on data sets of time series: prefer the mean scaled score](https://mlsanity.com/paper/2606.24715/) (2026)
- [Invariant Probabilistic Prediction](https://mlsanity.com/paper/2309.10083/) (2023)
- [CNN-based Surface Temperature Forecasts with Ensemble Numerical Weather Prediction](https://mlsanity.com/paper/2507.18937/) (2025)

## Abstract (arXiv)

Averages of proper scoring rules are often used to rank probabilistic forecasts. In many cases, the individual terms in these averages are based on observations and forecasts from different distributions. We show that some of the most popular proper scoring rules, such as the continuous ranked probability score (CRPS), give more importance to observations with large uncertainty which can lead to unintuitive rankings. To describe this issue, we define the concept of local scale invariance for scoring rules. A new class of generalized proper kernel scoring rules is derived and as a member of this class we propose the scaled CRPS (SCRPS). This new proper scoring rule is locally scale invariant and therefore works in the case of varying uncertainty. Like CRPS it is computationally available for output from ensemble forecasts, and does not require the ability to evaluate densities of forecasts. We further define robustness of scoring rules, show why this also is an important concept for average scores, and derive new proper scoring rules that are robust against outliers. The theoretical findings are illustrated in three different applications from spatial statistics, stochastic volatility models, and regression for count data.