# Statistical comparison of classifiers through Bayesian hierarchical modelling

arXiv: https://arxiv.org/abs/1609.08905 · HTML: https://mlsanity.com/paper/1609.08905/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2016.

## Built on by (most cited first, 14 of 14)

- [An Approach to Multiple Comparison Benchmark Evaluations that is Stable Under Manipulation of the Comparate Set](https://mlsanity.com/paper/2305.11921/) (2023)
- [Statistical Comparisons of Classifiers by Generalized Stochastic Dominance](https://mlsanity.com/paper/2209.01857/) (2022)
- [Robust Statistical Comparison of Random Variables with Locally Varying Scale of Measurement](https://mlsanity.com/paper/2306.12803/) (2023)
- [A Comparative Evaluation of Quantification Methods](https://mlsanity.com/paper/2103.03223/) (2021)
- [Statistical Multicriteria Benchmarking via the GSD-Front](https://mlsanity.com/paper/2406.03924/) (2024)
- [A Bayesian Bradley-Terry model to compare multiple ML algorithms on multiple data sets](https://mlsanity.com/paper/2208.04935/) (2022)
- [Handling Imbalanced Classification Problems With Support Vector Machines via Evolutionary Bilevel Optimization](https://mlsanity.com/paper/2204.10231/) (2022)
- [Learning Choice Functions with Gaussian Processes](https://mlsanity.com/paper/2302.00406/) (2023)
- [Fusion or Confusion? Multimodal Complexity Is Not All You Need](https://mlsanity.com/paper/2512.22991/) (2025)
- [Separating Rule Discovery and Global Solution Composition in a Learning Classifier System](https://mlsanity.com/paper/2202.01677/) (2022)
- [Investigating the Impact of Independent Rule Fitnesses in a Learning Classifier System](https://mlsanity.com/paper/2207.05582/) (2022)
- [Empirical Comparison between Cross-Validation and Mutation-Validation in Model Selection](https://mlsanity.com/paper/2311.14079/) (2023)
- [Generalizability of experimental studies](https://mlsanity.com/paper/2406.17374/) (2024)
- [Kernel-based External Validity of Experimental Studies](https://mlsanity.com/paper/2610.13625/) (2026)

## Abstract (arXiv)

Usually one compares the accuracy of two competing classifiers via null hypothesis significance tests (nhst). Yet the nhst tests suffer from important shortcomings, which can be overcome by switching to Bayesian hypothesis testing. We propose a Bayesian hierarchical model which jointly analyzes the cross-validation results obtained by two classifiers on multiple data sets. It returns the posterior probability of the accuracies of the two classifiers being practically equivalent or significantly different. A further strength of the hierarchical model is that, by jointly analyzing the results obtained on all data sets, it reduces the estimation error compared to the usual approach of averaging the cross-validation results obtained on a given data set.