# Binary Quantification and Dataset Shift: An Experimental Investigation

arXiv: https://arxiv.org/abs/2310.04565 · HTML: https://mlsanity.com/paper/2310.04565/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 66 references that are in the map.
- First posted on arXiv 6 Oct 2023.

## Builds on (most cited first, 19 of 19)

- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Regularized Learning for Domain Adaptation under Label Shifts](https://mlsanity.com/paper/1903.09734/) (2019)
- [Challenges in Benchmarking Stream Learning Algorithms with Real-world Data](https://mlsanity.com/paper/2005.00113/) (2020)
- [Fisher consistency for prior probability shift](https://mlsanity.com/paper/1701.05512/) (2017)
- [Estimating and Explaining Model Performance When Both Covariates and Labels Shift](https://mlsanity.com/paper/2209.08436/) (2022)
- [Quantification under prior probability shift: the ratio estimator and its extensions](https://mlsanity.com/paper/1807.03929/) (2018)
- [QuaPy: A Python-Based Framework for Quantification](https://mlsanity.com/paper/2106.11057/) (2021)
- [Evaluation Measures for Quantification: An Axiomatic Approach](https://mlsanity.com/paper/1809.01991/) (2018)
- [The Hitchhiker's Guide to Prior-Shift Adaptation](https://mlsanity.com/paper/2106.11695/) (2021)
- [A Comparative Evaluation of Quantification Methods](https://mlsanity.com/paper/2103.03223/) (2021)
- [Class Prior Estimation under Covariate Shift: No Problem?](https://mlsanity.com/paper/2206.02449/) (2022)
- [A Recurrent Neural Network for Sentiment Quantification](https://mlsanity.com/paper/1809.00836/) (2018)
- [Re-Assessing the "Classify and Count" Quantification Method](https://mlsanity.com/paper/2011.02552/) (2020)
- [Tweet Sentiment Quantification: An Experimental Re-Evaluation](https://mlsanity.com/paper/2011.08091/) (2020)
- [Optimizing Text Quantifiers for Multivariate Loss Functions](https://mlsanity.com/paper/1502.05491/) (2015)
- [Quantification in-the-wild: data-sets and baselines](https://mlsanity.com/paper/1510.04811/) (2015)

## Built on by (most cited first, 6 of 6)

- [On the Interconnections of Calibration, Quantification, and Classifier Accuracy Prediction under Dataset Shift](https://mlsanity.com/paper/2505.11380/) (2025)
- [Adjusted Count Quantification Learning on Graphs](https://mlsanity.com/paper/2503.09395/) (2025)
- [Efficient quantification on large-scale networks](https://mlsanity.com/paper/2503.15267/) (2025)
- [Distribution Matching for Graph Quantification Under Structural Covariate Shift](https://mlsanity.com/paper/2601.00864/) (2025)
- [Quantifying Feature Importance for Online Content Moderation](https://mlsanity.com/paper/2510.19882/) (2025)
- [Class-Conditioned Gaussian Mixture Modeling for Imbalanced Time Series Quantification](https://mlsanity.com/paper/2608.21473/) (2026)

## Abstract (arXiv)

Quantification is the supervised learning task that consists of training predictors of the class prevalence values of sets of unlabelled data, and is of special interest when the labelled data on which the predictor has been trained and the unlabelled data are not IID, i.e., suffer from dataset shift. To date, quantification methods have mostly been tested only on a special case of dataset shift, i.e., prior probability shift; the relationship between quantification and other types of dataset shift remains, by and large, unexplored. In this work we carry out an experimental analysis of how current quantification algorithms behave under different types of dataset shift, in order to identify limitations of current approaches and hopefully pave the way for the development of more broadly applicable methods. We do this by proposing a fine-grained taxonomy of types of dataset shift, by establishing protocols for the generation of datasets affected by these types of shift, and by testing existing quantification methods on the datasets thus generated. One finding that results from this investigation is that many existing quantification methods that had been found robust to prior probability shift are not necessarily robust to other types of dataset shift. A second finding is that no existing quantification method seems to be robust enough to dealing with all the types of dataset shift we simulate in our experiments. The code needed to reproduce all our experiments is publicly available at https://github.com/pglez82/quant_datasetshift.