# Evaluation Metrics as Averaged Outcomes of Fair Gambles

arXiv: https://arxiv.org/abs/2401.14483 · HTML: https://mlsanity.com/paper/2401.14483/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jan 2024.

## Built on by (most cited first, 2 of 2)

- [Scoring Rules and Calibration for Imprecise Probabilities](https://mlsanity.com/paper/2410.23001/) (2024)
- [On Calibration in Multi-Distribution Learning](https://mlsanity.com/paper/2412.14142/) (2024)

## Abstract (arXiv)

In the current practices of machine learning, the evaluation of forecasts has become a cornerstone of scientific progress. A multitude of evaluation metrics have been suggested and used to qualify "good" forecasts. What do those metrics share? How are they related? In this work, we use a protocol borrowed from game-theoretic probability to show that a large part of evaluation metrics can be viewed as averaged outcomes of fair gambles. Intuitively, a fair gambler is one which a forecaster would expect to fail. Hence, the gambler's ability to gain disproves the quality of the forecast. Standard evaluation metrics are then variants of choices of such fair gambles. In particular, this choice is structured along two dimensions, one of which separates calibration-type and regret-type metrics. In particular, this framework sheds light on the relationship of calibration and regret showing a theoretical equivalence in their ability to evaluate when being scaled appropriately, but the incomparability of obtained scores.