# Model Comparison and Calibration Assessment: User Guide for Consistent Scoring Functions in Machine Learning and Actuarial Practice

arXiv: https://arxiv.org/abs/2202.12780 · HTML: https://mlsanity.com/paper/2202.12780/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 83 references that are in the map.
- First posted on arXiv 25 Feb 2022.

## Builds on (most cited first, 21 of 21)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1412.6614/) (2014)
- [Two models of double descent for weak features](https://mlsanity.com/paper/1903.07571/) (2019)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://mlsanity.com/paper/1811.12808/) (2018)
- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning](https://mlsanity.com/paper/2109.02355/) (2021)
- [Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination](https://mlsanity.com/paper/2108.03210/) (2021)
- [Of Quantiles and Expectiles: Consistent Scoring Functions, Choquet Representations, and Forecast Rankings](https://mlsanity.com/paper/1503.08195/) (2015)
- [Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited](https://mlsanity.com/paper/2008.03033/) (2020)
- [Forecaster's Dilemma: Extreme Events and Forecast Evaluation](https://mlsanity.com/paper/1512.09244/) (2015)
- [Generic Conditions for Forecast Dominance](https://mlsanity.com/paper/1805.09902/) (2018)
- [Incentive-Compatible Forecasting Competitions](https://mlsanity.com/paper/2101.01816/) (2021)
- [The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation](https://mlsanity.com/paper/2005.01835/) (2020)
- [Forecast Evaluation of Quantiles, Prediction Intervals, and other Set-Valued Functionals](https://mlsanity.com/paper/1910.07912/) (2019)
- [Valid sequential inference on probability forecast performance](https://mlsanity.com/paper/2103.08402/) (2021)
- [Impartial Predictive Modeling and the Use of Proxy Variables](https://mlsanity.com/paper/1608.00528/) (2016)
- [Is the mode elicitable relative to unimodal distributions?](https://mlsanity.com/paper/2109.00464/) (2021)

## Built on by (most cited first, 13 of 13)

- [Beyond IID: How General Are Tabular Foundation Models, Really?](https://mlsanity.com/paper/2606.30410/) (2026)
- [Neural networks for insurance pricing with frequency and severity data: a benchmark study from data preprocessing to technical tariff](https://mlsanity.com/paper/2310.12671/) (2023)
- [Actuarial Applications of Natural Language Processing Using Transformers: Case Studies for Using Text Features in an Actuarial Context](https://mlsanity.com/paper/2206.02014/) (2022)
- [Ensemble learning for uncertainty estimation with application to the correction of satellite precipitation products](https://mlsanity.com/paper/2403.10567/) (2024)
- [From Point to probabilistic gradient boosting for claim frequency and severity prediction](https://mlsanity.com/paper/2412.14916/) (2024)
- [Calibration Bands for Mean Estimates within the Exponential Dispersion Family](https://mlsanity.com/paper/2503.18896/) (2025)
- [Assessing model calibration with boosting trees](https://mlsanity.com/paper/2606.08084/) (2026)
- [Global quantitative robustness of regression feed-forward neural networks](https://mlsanity.com/paper/2211.10124/) (2022)
- [Towards a data-driven debt collection strategy based on an advanced machine learning framework](https://mlsanity.com/paper/2311.06292/) (2023)
- [Variable transformations in consistent loss functions](https://mlsanity.com/paper/2502.16542/) (2025)
- [Explainable Boosting Machine for Predicting Claim Severity and Frequency in Car Insurance](https://mlsanity.com/paper/2503.21321/) (2025)
- [Every pooling rule has its world: matching probability combination rules to situations and stakes](https://mlsanity.com/paper/2608.11275/) (2026)
- [A Practical Guide on Graphical Model Validation](https://mlsanity.com/paper/2609.26445/) (2026)

## Abstract (arXiv)

One of the main tasks of actuaries and data scientists is to build good predictive models for certain phenomena such as the claim size or the number of claims in insurance. These models ideally exploit given feature information to enhance the accuracy of prediction. This user guide revisits and clarifies statistical techniques to assess the calibration or adequacy of a model on the one hand, and to compare and rank different models on the other hand. In doing so, it emphasises the importance of specifying the prediction target functional at hand a priori (e.g. the mean or a quantile) and of choosing the scoring function in model comparison in line with this target functional. Guidance for the practical choice of the scoring function is provided. Striving to bridge the gap between science and daily practice in application, it focuses mainly on the pedagogical presentation of existing results and of best practice. The results are accompanied and illustrated by two real data case studies on workers' compensation and customer churn.