# Predicting Out-of-Distribution Error with Confidence Optimal Transport

arXiv: https://arxiv.org/abs/2302.05018 · HTML: https://mlsanity.com/paper/2302.05018/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 21 references that are in the map.
- First posted on arXiv 10 Feb 2023.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022)
- [Assessing Generalization of SGD via Disagreement](https://mlsanity.com/paper/2106.13799/) (2021)
- [Predicting with Confidence on Unseen Distributions](https://mlsanity.com/paper/2107.03315/) (2021)
- [Are Labels Always Necessary for Classifier Accuracy Evaluation?](https://mlsanity.com/paper/2007.02915/) (2020)
- [Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift](https://mlsanity.com/paper/2206.13089/) (2022)
- [Predicting Out-of-Distribution Error with the Projection Norm](https://mlsanity.com/paper/2202.05834/) (2022)
- [Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approach](https://mlsanity.com/paper/1705.07086/) (2017)

## Built on by (most cited first, 13 of 13)

- [The Entropy Enigma: Success and Failure of Entropy Minimization](https://mlsanity.com/paper/2405.05012/) (2024)
- [GNNEvaluator: Evaluating GNN Performance On Unseen Graphs Without Labels](https://mlsanity.com/paper/2310.14586/) (2023)
- [Energy-based Automated Model Evaluation](https://mlsanity.com/paper/2401.12689/) (2024)
- [(Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancy](https://mlsanity.com/paper/2306.00312/) (2023)
- [Chasing Fairness Under Distribution Shift: A Model Weight Perturbation Approach](https://mlsanity.com/paper/2303.03300/) (2023)
- [Monitoring Risks in Test-Time Adaptation](https://mlsanity.com/paper/2507.08721/) (2025)
- [Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings](https://mlsanity.com/paper/2505.22356/) (2025)
- [Distance Matters For Improving Performance Estimation Under Covariate Shift](https://mlsanity.com/paper/2308.07223/) (2023)
- [Beyond Top-Class Agreement: Using Divergences to Forecast Performance under Distribution Shift](https://mlsanity.com/paper/2312.08033/) (2023)
- [Source-Free Domain-Invariant Performance Prediction](https://mlsanity.com/paper/2408.02209/) (2024)
- [Poor-Supervised Evaluation for SuperLLM via Mutual Consistency](https://mlsanity.com/paper/2408.13738/) (2024)
- [Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks](https://mlsanity.com/paper/2412.20682/) (2024)
- [ODD: Overlap-aware Estimation of Model Performance under Distribution Shift](https://mlsanity.com/paper/2506.14978/) (2025)

## Abstract (arXiv)

Out-of-distribution (OOD) data poses serious challenges in deployed machine learning models as even subtle changes could incur significant performance drops. Being able to estimate a model's performance on test data is important in practice as it indicates when to trust to model's decisions. We present a simple yet effective method to predict a model's performance on an unknown distribution without any addition annotation. Our approach is rooted in the Optimal Transport theory, viewing test samples' output softmax scores from deep neural networks as empirical samples from an unknown distribution. We show that our method, Confidence Optimal Transport (COT), provides robust estimates of a model's performance on a target domain. Despite its simplicity, our method achieves state-of-the-art results on three benchmark datasets and outperforms existing methods by a large margin.