mlsanityData to 7 Oct 2026

Detecting and Correcting for Label Shift with Black Box Predictors

arXiv 1802.03916 ↗First posted 12 Feb 2018cs.LG cs.AI cs.NEcited by 364

Not on the map. Why?

Abstract (arXiv)

Faced with distribution shift between training and test set, we wish to detect and quantify the shift, and to correct our classifiers without test set labels. Motivated by medical diagnosis, where diseases (targets) cause symptoms (observations), we focus on label shift, where the label marginal p(y) changes but the conditional p(x| y) does not. We propose Black Box Shift Estimation (BBSE) to estimate the test distribution p(y). BBSE exploits arbitrary black box predictors to reduce dimensionality prior to shift correction. While better predictors give tighter estimates, BBSE works even when predictors are biased, inaccurate, or uncalibrated, so long as their confusion matrices are invertible. We prove BBSE's consistency, bound its error, and introduce a statistical test that uses BBSE to detect shift. We also leverage BBSE to correct classifiers. Experiments demonstrate accurate estimates and improved prediction, even on high-dimensional datasets of natural images.

Key facts. Cited by 364 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 57; 2022: 65; 2023: 77; 2024: 48; 2025: 55; 2026: 62.

364 papers.

Sort:

By area:

  1. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  2. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  3. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  4. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
  5. 2023Prediction-Powered Inferencecited by 214
  6. 2021Towards Causal Representation Learningcited by 207
  7. 2023Data-centric Artificial Intelligence: A Surveycited by 202
  8. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 141
  9. 2021Model-Based Domain Generalizationcited by 103
  10. 2021Distribution-free uncertainty quantification for classification under label shiftcited by 99
  11. 2022Deep Unsupervised Domain Adaptation: A Review of Recent Advances and Perspectivescited by 98
  12. 2021Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognitioncited by 95
38 more
  1. 2022From Concept Drift to Model Degradation: An Overview on Performance-Aware Drift Detectorscited by 82
  2. 2021Adaptive Methods for Real-World Domain Generalizationcited by 66
  3. 2022Probable Domain Generalization via Quantile Risk Minimizationcited by 66
  4. 2021A Prototype-Oriented Framework for Unsupervised Domain Adaptationcited by 62
  5. 2023Domain Adaptation for Time Series Under Feature and Label Shiftscited by 60
  6. 2022Data Feedback Loops: Model-driven Amplification of Dataset Biasescited by 59
  7. 2021Mandoline: Model Evaluation under Distribution Shiftcited by 55
  8. 2021Meaningfully Debugging Model Mistakes using Conceptual Counterfactual Explanationscited by 53
  9. 2021Mixture Proportion Estimation and PU Learning: A Modern Approachcited by 52
  10. 2021DINE: Domain Adaptation from Single and Multiple Black-box Predictorscited by 51
  11. 2021Domain Adversarial Neural Networks for Domain Generalization: When It Works and How to Improvecited by 46
  12. 2022Diagnosing failures of fairness transfer across distribution shift in real-world medical settingscited by 45
  13. 2023Fusing Models with Complementary Expertisecited by 45
  14. 2023MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoningcited by 45
  15. 2021Online Adaptation to Label Distribution Shiftcited by 44
  16. 2022ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasetscited by 44
  17. 2021Adversarial Unsupervised Domain Adaptation with Conditional and Label Shift: Infer, Align and Iteratecited by 43
  18. 2021PAC Prediction Sets Under Covariate Shiftcited by 42
  19. 2021Near-Optimal Linear Regression under Distribution Shiftcited by 40
  20. 2021Understanding the role of importance weighting for deep learningcited by 38
  21. 2023Diagnosing Model Performance Under Distribution Shiftcited by 37
  22. 2023Subspace Identification for Multi-Source Domain Adaptationcited by 36
  23. 2021Sinkhorn Label Allocation: Semi-Supervised Classification via Annealed Self-Trainingcited by 32
  24. 2022Domain Adaptation under Open Set Label Shiftcited by 32
  25. 2023Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Modelscited by 32
  26. 2021An Online Learning Approach to Interpolation and Extrapolation in Domain Generalizationcited by 30
  27. 2021Cross-Domain Label-Adaptive Stance Detectioncited by 30
  28. 2021On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Studycited by 30
  29. 2022Calibrated Selective Classificationcited by 30
  30. 2023RLSbench: Domain Adaptation Under Relaxed Label Shiftcited by 30
  31. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  32. 2023Conformal Prediction for Federated Uncertainty Quantification Under Label Shiftcited by 29
  33. 2022Adapting to Online Label Shift with Provable Guaranteescited by 26
  34. 2022Towards Explaining Distribution Shiftscited by 26
  35. 2024Pairwise Alignment Improves Graph Domain Adaptationcited by 25
  36. 2021Aggregating From Multiple Target-Shifted Sourcescited by 24
  37. 2023Calibration of Time-Series Forecasting: Detecting and Adapting Context-Driven Distribution Shiftcited by 23
  38. 2023Adaptive Test-Time Personalization for Federated Learningcited by 23

50 most cited of 364 All 364 (JSON)