# A Comparative Evaluation of Quantification Methods

arXiv: https://arxiv.org/abs/2103.03223 · HTML: https://mlsanity.com/paper/2103.03223/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 52 references that are in the map.
- First posted on arXiv 4 Mar 2021.

## Builds on (most cited first, 10 of 10)

- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis](https://mlsanity.com/paper/1606.04316/) (2016)
- [Fisher consistency for prior probability shift](https://mlsanity.com/paper/1701.05512/) (2017)
- [Statistical comparison of classifiers through Bayesian hierarchical modelling](https://mlsanity.com/paper/1609.08905/) (2016)
- [Evaluation Measures for Quantification: An Axiomatic Approach](https://mlsanity.com/paper/1809.01991/) (2018)
- [A Recurrent Neural Network for Sentiment Quantification](https://mlsanity.com/paper/1809.00836/) (2018)
- [Unified Framework for Quantification](https://mlsanity.com/paper/1606.00868/) (2016)
- [Re-Assessing the "Classify and Count" Quantification Method](https://mlsanity.com/paper/2011.02552/) (2020)
- [Confidence intervals for class prevalences under prior probability shift](https://mlsanity.com/paper/1906.04119/) (2019)
- [Does quantification without adjustments work?](https://mlsanity.com/paper/1602.08780/) (2016)

## Built on by (most cited first, 11 of 11)

- [QuaPy: A Python-Based Framework for Quantification](https://mlsanity.com/paper/2106.11057/) (2021)
- [Accurate and Well-Calibrated ICD Code Assignment Through Attention Over Diverse Label Embeddings](https://mlsanity.com/paper/2402.03172/) (2024)
- [Predictable Artificial Intelligence](https://mlsanity.com/paper/2310.06167/) (2023)
- [Binary Quantification and Dataset Shift: An Experimental Investigation](https://mlsanity.com/paper/2310.04565/) (2023)
- [On the Interconnections of Calibration, Quantification, and Classifier Accuracy Prediction under Dataset Shift](https://mlsanity.com/paper/2505.11380/) (2025)
- [Estimating prevalence with precision and accuracy](https://mlsanity.com/paper/2507.06061/) (2025)
- [Multi-Label Quantification](https://mlsanity.com/paper/2211.08063/) (2022)
- [Continuous Sweep for Binary Quantification Learning](https://mlsanity.com/paper/2308.08387/) (2023)
- [Comments on Friedman's Method for Class Distribution Estimation](https://mlsanity.com/paper/2405.16666/) (2024)
- [AuAu: A Benchmark for Auditing Authoritarian Alignment in Large Language Models](https://mlsanity.com/paper/2606.16127/) (2026)
- [Geometry-Aware Bayesian Quantification via Compositional Data Analysis](https://mlsanity.com/paper/2607.04977/) (2026)

## Abstract (arXiv)

Quantification represents the problem of estimating the distribution of class labels on unseen data. It also represents a growing research field in supervised machine learning, for which a large variety of different algorithms has been proposed in recent years. However, a comprehensive empirical comparison of quantification methods that supports algorithm selection is not available yet. In this work, we close this research gap by conducting a thorough empirical performance comparison of 24 different quantification methods on overall more than 40 data sets, considering binary as well as multiclass quantification settings. We observe that no single algorithm generally outperforms all competitors, but identify a group of methods including the threshold selection-based Median Sweep and TSMax methods, the DyS framework including the HDy method, Forman's mixture model, and Friedman's method that performs best in the binary setting. For the multiclass setting, we observe that a different, broad group of algorithms yields good performance, including the HDx method, the Generalized Probabilistic Adjusted Count, the readme method, the energy distance minimization method, the EM algorithm for quantification, and Friedman's method. We also find that tuning the underlying classifiers has in most cases only a limited impact on the quantification performance. More generally, we find that the performance on multiclass quantification is inferior to the results obtained in the binary setting. Our results can guide practitioners who intend to apply quantification algorithms and help researchers to identify opportunities for future research.