mlsanityData to 14 Oct 2026

On the Interconnections of Calibration, Quantification, and Classifier Accuracy Prediction under Dataset Shift

arXiv 2505.11380 ↗First posted 16 May 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

When the distribution of the data used to train a classifier differs from that of the test data, i.e., under dataset shift, well-established routines for calibrating the decision scores of the classifier, estimating the proportion of positives in a test sample, or estimating the accuracy of the classifier, become particularly challenging. This paper investigates the interconnections among three fundamental problems, calibration, quantification, and classifier accuracy prediction, under dataset shift conditions. Specifically, we prove their equivalence through mutual reduction, i.e., we show that access to an oracle for any one of these tasks enables the resolution of the other two. Based on these proofs, we propose new methods for each problem based on direct adaptations of well-established methods borrowed from the other disciplines. Our results show such methods are often competitive, and sometimes even surpass the performance of dedicated approaches from each discipline. The main goal of this paper is to fostering cross-fertilization among these research areas, encouraging the development of unified approaches and promoting synergies across the fields.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 38 of its 78 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

38 of 78 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  8. 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 366
  9. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  10. 2018Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shiftcited by 243
  11. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 145
  12. 2019Regularized Learning for Domain Adaptation under Label Shiftscited by 120
26 more
  1. 2020A Unified View of Label Shift Estimationcited by 119
  2. 2021Assessing Generalization of SGD via Disagreementcited by 117
  3. 2015Should we really use post-hoc tests based on mean-ranks?cited by 116
  4. 2021Predicting with Confidence on Unseen Distributionscited by 116
  5. 2021Soft Calibration Objectives for Neural Networkscited by 92
  6. 2021Classifier Calibration: A survey on how to assess and improve predicted class probabilitiescited by 73
  7. 2021Mandoline: Model Evaluation under Distribution Shiftcited by 55
  8. 2021Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensemblescited by 54
  9. 2020Calibrated Prediction with Covariate Shift via Unsupervised Domain Adaptationcited by 43
  10. 2020Transferable Calibration with Lower Bias and Variance in Domain Adaptationcited by 41
  11. 2021QuaPy: A Python-Based Framework for Quantificationcited by 14
  12. 2018Evaluation Measures for Quantification: An Axiomatic Approachcited by 13
  13. 2023Transfer Knowledge from Head to Tail: Uncertainty Calibration under Long-tailed Distributioncited by 12
  14. 2021A Comparative Evaluation of Quantification Methodscited by 11
  15. 2021Calibrating sufficientlycited by 10
  16. 2022Class Prior Estimation under Covariate Shift: No Problem?cited by 10
  17. 2023Kernel Density Estimation for Multiclass Quantificationcited by 8
  18. 2016Unified Framework for Quantificationcited by 7
  19. 2020Tweet Sentiment Quantification: An Experimental Re-Evaluationcited by 7
  20. 2015Quantification in-the-wild: data-sets and baselinescited by 6
  21. 2023Binary Quantification and Dataset Shift: An Experimental Investigationcited by 6
  22. 2024Calibrate-Extrapolate: Rethinking Prevalence Estimation with Black Box Classifierscited by 6
  23. 2023Label Shift Quantification with Robustness Guarantees via Distribution Feature Matchingcited by 5
  24. 2023Invariance assumptions for class distribution estimationcited by 4
  25. 2022Minimizing Uncertainty in Prevalence Estimatescited by 3
  26. 2024Quantification using Permutation-Invariant Networks based on Histogramscited by 3

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Recalibrating binary probabilistic classifierscited by 1
  2. 2025Quantifying Feature Importance for Online Content Moderationcited by 0