# Performance Prediction Under Dataset Shift

arXiv: https://arxiv.org/abs/2206.10697 · HTML: https://mlsanity.com/paper/2206.10697/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 21 Jun 2022.

## Builds on (most cited first, 13 of 13)

- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Measuring Robustness to Natural Distribution Shifts in Image Classification](https://mlsanity.com/paper/2007.00644/) (2020)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022)
- [Predicting with Confidence on Unseen Distributions](https://mlsanity.com/paper/2107.03315/) (2021)
- [Are Labels Always Necessary for Classifier Accuracy Evaluation?](https://mlsanity.com/paper/2007.02915/) (2020)
- [Towards Robust and Reliable Algorithmic Recourse](https://mlsanity.com/paper/2102.13620/) (2021)
- [What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?](https://mlsanity.com/paper/2106.05961/) (2021)
- [Confidence Scoring Using Whitebox Meta-models with Linear Classifier Probes](https://mlsanity.com/paper/1805.05396/) (2018)
- [Learning Prediction Intervals for Model Performance](https://mlsanity.com/paper/2012.08625/) (2020)
- [The Data Representativeness Criterion: Predicting the Performance of Supervised Classification Based on Data Set Similarity](https://mlsanity.com/paper/2002.12105/) (2020)
- [Ensembling Shift Detectors: an Extensive Empirical Evaluation](https://mlsanity.com/paper/2106.14608/) (2021)

## Built on by (most cited first, 3 of 3)

- [Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings](https://mlsanity.com/paper/2505.22356/) (2025)
- [Detecting Domain Shift in Multiple Instance Learning for Digital Pathology Using Fr'echet Domain Distance](https://mlsanity.com/paper/2405.09934/) (2024)
- [Distance Matters For Improving Performance Estimation Under Covariate Shift](https://mlsanity.com/paper/2308.07223/) (2023)

## Abstract (arXiv)

ML models deployed in production often have to face unknown domain changes, fundamentally different from their training settings. Performance prediction models carry out the crucial task of measuring the impact of these changes on model performance. We study the generalization capabilities of various performance prediction models to new domains by learning on generated synthetic perturbations. Empirical validation on a benchmark of ten tabular datasets shows that models based upon state-of-the-art shift detection metrics are not expressive enough to generalize to unseen domains, while Error Predictors bring a consistent improvement in performance prediction under shift. We additionally propose a natural and effortless uncertainty estimation of the predicted accuracy that ensures reliable use of performance predictors. Our implementation is available at https: //github.com/dataiku-research/performance_prediction_under_shift.