Diagnostic Tool for Out-of-Sample Model Evaluation
Not on the map. Why?
Abstract (arXiv)
Assessment of model fitness is a key part of machine learning. The standard paradigm is to learn models by minimizing a chosen loss function averaged over training data, with the aim of achieving small losses on future data. In this paper, we consider the use of a finite calibration data set to characterize the future, out-of-sample losses of a model. We propose a simple model diagnostic tool that provides finite-sample guarantees under weak assumptions. The tool is simple to compute and to interpret. Several numerical experiments are presented to show how the proposed method quantifies the impact of distribution shifts, aids the analysis of regression, and enables model selection as well as hyper-parameter tuning.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 11 of its 61 references that are in the map. First posted on arXiv 22 Jun 2022.
Builds on
11 of 61 references.
References · 11
- 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
- 2016Distribution-Free Predictive Inference For Regressioncited by 657
- 2019Conformal Prediction Under Covariate Shiftcited by 511
- 2020Learning under Concept Drift: A Reviewcited by 470
- 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 364
- 2018Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shiftcited by 240
- 2020Estimating means of bounded random variables by bettingcited by 164
- 2020Tilted Empirical Risk Minimizationcited by 103
- 2018Model Selection Techniques -- An Overviewcited by 36
- 2021Reliable and Trustworthy Machine Learning for Health Using Dataset Shift Detectioncited by 21
- 2018Data Consistency Approach to Model Validationcited by 1
Built on by
Not cited yet.