# Learning Prediction Intervals for Model Performance

arXiv: https://arxiv.org/abs/2012.08625 · HTML: https://mlsanity.com/paper/2012.08625/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2020.

## Built on by (most cited first, 9 of 9)

- [Post-hoc Uncertainty Learning using a Dirichlet Meta-Model](https://mlsanity.com/paper/2212.07359/) (2022)
- [Uncertainty Quantification 360: A Holistic Toolkit for Quantifying and Communicating the Uncertainty of AI](https://mlsanity.com/paper/2106.01410/) (2021)
- [Performance Prediction Under Dataset Shift](https://mlsanity.com/paper/2206.10697/) (2022)
- [Detecting Domain Shift in Multiple Instance Learning for Digital Pathology Using Fr'echet Domain Distance](https://mlsanity.com/paper/2405.09934/) (2024)
- [SurvUnc: A Meta-Model Based Uncertainty Quantification Framework for Survival Analysis](https://mlsanity.com/paper/2505.14803/) (2025)
- [Uncertainty Aware Neural Network from Similarity and Sensitivity](https://mlsanity.com/paper/2304.14925/) (2023)
- [Uncertainty Quantification for Rule-Based Models](https://mlsanity.com/paper/2211.01915/) (2022)
- [MetaErr: Towards Predicting Error Patterns in Deep Neural Networks](https://mlsanity.com/paper/2604.23289/) (2026)
- [Bi-Level Chaotic Fusion Based Graph Convolutional Network for Stock Market Prediction Interval](https://mlsanity.com/paper/2605.16324/) (2026)

## Abstract (arXiv)

Understanding model performance on unlabeled data is a fundamental challenge of developing, deploying, and maintaining AI systems. Model performance is typically evaluated using test sets or periodic manual quality assessments, both of which require laborious manual data labeling. Automated performance prediction techniques aim to mitigate this burden, but potential inaccuracy and a lack of trust in their predictions has prevented their widespread adoption. We address this core problem of performance prediction uncertainty with a method to compute prediction intervals for model performance. Our methodology uses transfer learning to train an uncertainty model to estimate the uncertainty of model performance predictions. We evaluate our approach across a wide range of drift conditions and show substantial improvement over competitive baselines. We believe this result makes prediction intervals, and performance prediction in general, significantly more practical for real-world use.