mlsanityData to 7 Oct 2026

Leveraging Unlabeled Data to Predict Out-of-Distribution Performance

arXiv 2201.04234 ↗First posted 11 Jan 2022cs.LG stat.MLcited by 141

Not on the map. Why?

Abstract (arXiv)

Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions that may cause performance drops. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples for which model confidence exceeds that threshold. ATC outperforms previous methods across several model architectures, types of distribution shifts (e.g., due to synthetic corruptions, dataset reproduction, or novel subpopulations), and datasets (Wilds, ImageNet, Breeds, CIFAR, and MNIST). In our experiments, ATC estimates target performance 2-4× more accurately than prior methods. We also explore the theoretical foundations of the problem, proving that, in general, identifying the accuracy is just as hard as identifying the optimal predictor and thus, the efficacy of any method rests upon (perhaps unstated) assumptions on the nature of the shift. Finally, analyzing our method on some toy distributions, we provide insights concerning when it works. Code is available at https://github.com/saurabhgarg1996/ATCcode/.

Key facts. Cited by 141 papers in the map (data through 7 Oct 2026). Builds on 51 of its 85 references that are in the map. First posted on arXiv 11 Jan 2022.

Builds on

51 of 85 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2017On Calibration of Modern Neural Networkscited by 4,860
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  11. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
38 more
  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  4. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  6. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  7. 2017Selective Classification for Deep Neural Networkscited by 799
  8. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  9. 2017Exploring Generalization in Deep Learningcited by 595
  10. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  11. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  12. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  13. 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 364
  14. 2017Functional Map of the Worldcited by 338
  15. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  16. 2018Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shiftcited by 240
  17. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  18. 2018To Trust Or Not To Trust A Classifiercited by 205
  19. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 197
  20. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  21. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  22. 2018Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approachcited by 125
  23. 2020Domain Adaptation with Conditional Distribution Matching and Generalized Label Shiftcited by 124
  24. 2020A Unified View of Label Shift Estimationcited by 119
  25. 2019Regularized Learning for Domain Adaptation under Label Shiftscited by 118
  26. 2021Assessing Generalization of SGD via Disagreementcited by 116
  27. 2021Predicting with Confidence on Unseen Distributionscited by 116
  28. 2020Hybrid Models for Open Set Recognitioncited by 109
  29. 2020Are Labels Always Necessary for Classifier Accuracy Evaluation?cited by 104
  30. 2017Implicit Regularization in Deep Learningcited by 90
  31. 2021What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?cited by 63
  32. 2019Cold Case: The Lost MNIST Digitscited by 61
  33. 2019Generalization bounds for deep convolutional neural networkscited by 58
  34. 2020Estimating Generalization under Distribution Shifts via Domain-Invariant Representationscited by 56
  35. 2021Mandoline: Model Evaluation under Distribution Shiftcited by 55
  36. 2021Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensemblescited by 54
  37. 2019Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resiliencecited by 51
  38. 2017Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approachcited by 27

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 22; 2023: 42; 2024: 33; 2025: 23; 2026: 21.

141 papers.

Sort:

By area:

  1. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  2. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  3. 2024AutoEval Done Right: Using Synthetic Data for Model Evaluationcited by 53
  4. 2022RankFeat: Rank-1 Feature Removal for Out-of-distribution Detectioncited by 52
  5. 2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
  6. 2022Domain Adaptation under Open Set Label Shiftcited by 32
  7. 2023Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Datacited by 31
  8. 2023RLSbench: Domain Adaptation Under Relaxed Label Shiftcited by 30
  9. 2024The Entropy Enigma: Success and Failure of Entropy Minimizationcited by 24
  10. 2022On the Strong Correlation Between Model Invariance and Generalizationcited by 23
  11. 2023Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimationcited by 22
  12. 2024What Do Learning Dynamics Reveal About Generalization in LLM Reasoning?cited by 22
38 more
  1. 2023Characterizing Out-of-Distribution Error via Optimal Transportcited by 20
  2. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 20
  3. 2022Deconstructing Distributions: A Pointwise Framework of Learningcited by 19
  4. 2022Estimating and Explaining Model Performance When Both Covariates and Labels Shiftcited by 19
  5. 2022A Learning Based Hypothesis Test for Harmful Covariate Shiftcited by 18
  6. 2023GNNEvaluator: Evaluating GNN Performance On Unseen Graphs Without Labelscited by 17
  7. 2022A Note on "Assessing Generalization of SGD via Disagreement"cited by 16
  8. 2023(Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancycited by 16
  9. 2024Energy-based Automated Model Evaluationcited by 16
  10. 2023On the Importance of Feature Separability in Predicting Out-Of-Distribution Errorcited by 15
  11. 2023Can We Evaluate Domain Adaptation Models Without Target-Domain Labels?cited by 14
  12. 2023Predicting Out-of-Distribution Error with Confidence Optimal Transportcited by 13
  13. 2024A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimationcited by 13
  14. 2023Demystifying Disagreement-on-the-Line in High Dimensionscited by 12
  15. 2023CIFAR-10-Warehouse: Broad and More Realistic Testbeds in Model Generalization Analysiscited by 12
  16. 2024Online GNN Evaluation Under Test-time Graph Distribution Shiftscited by 12
  17. 2022Understanding new tasks through the lens of training data via exponential tiltingcited by 11
  18. 2023A Bag-of-Prototypes Representation for Dataset-Level Applicationscited by 11
  19. 2023Online Label Shift: Optimal Dynamic Regret meets Practical Algorithmscited by 11
  20. 2023Test-Time Adaptation Induces Stronger Accuracy and Agreement-on-the-Linecited by 11
  21. 2022Evaluating Robustness to Dataset Shift via Parametric Robustness Setscited by 10
  22. 2022Estimating Model Performance under Domain Shifts with Class-Specific Confidence Scorescited by 10
  23. 2025Monitoring Risks in Test-Time Adaptationcited by 10
  24. 2022Monitoring Model Deterioration with Explainable Uncertainty Estimation via Non-parametric Bootstrapcited by 9
  25. 2022Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacitiescited by 9
  26. 2023Estimating Large Language Model Capabilities without Labeled Test Datacited by 9
  27. 2024Estimating Model Performance Under Covariate Shift Without Labelscited by 9
  28. 2024AETTA: Label-Free Accuracy Estimation for Test-Time Adaptationcited by 9
  29. 2024Sequential Harmful Shift Detection Without Labelscited by 9
  30. 2024Accurate and Scalable Estimation of Epistemic Uncertainty for Graph Neural Networkscited by 8
  31. 2024MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shiftscited by 8
  32. 2024Confidence-based Estimators for Predictive Performance in Model Monitoringcited by 8
  33. 2023ProbVLM: Probabilistic Adapter for Frozen Vision-Language Modelscited by 7
  34. 2023Selecting which Dense Retriever to use for Zero-Shot Searchcited by 7
  35. 2024Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitionscited by 7
  36. 2024Predicting the Performance of Foundation Models via Agreement-on-the-Linecited by 7
  37. 2022Explanation Shift: Detecting distribution shifts on tabular data via the explanation spacecited by 6
  38. 2023CAME: Contrastive Automated Model Evaluationcited by 6

50 most cited of 141 All 141 (JSON)