# Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis

arXiv: https://arxiv.org/abs/1606.04316 · HTML: https://mlsanity.com/paper/1606.04316/

## Key facts

- Cited by 93 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2016.

## Built on by (most cited first, 30 of 93)

- [DeepSMOTE: Fusing Deep Learning and SMOTE for Imbalanced Data](https://mlsanity.com/paper/2105.02340/) (2021)
- [MUSE-VAE: Multi-Scale VAE for Environment-Aware Long Term Trajectory Prediction](https://mlsanity.com/paper/2201.07189/) (2022)
- [How to avoid machine learning pitfalls: a guide for academic researchers](https://mlsanity.com/paper/2108.02497/) (2021)
- [An Approach to Multiple Comparison Benchmark Evaluations that is Stable Under Manipulation of the Comparate Set](https://mlsanity.com/paper/2305.11921/) (2023)
- [deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networks](https://mlsanity.com/paper/2204.06815/) (2022)
- [Zero-Shot Scene Graph Relation Prediction through Commonsense Knowledge Integration](https://mlsanity.com/paper/2107.05080/) (2021)
- [Statistical Comparisons of Classifiers by Generalized Stochastic Dominance](https://mlsanity.com/paper/2209.01857/) (2022)
- [STUDD: A Student-Teacher Method for Unsupervised Concept Drift Detection](https://mlsanity.com/paper/2103.00903/) (2021)
- [Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning](https://mlsanity.com/paper/2508.06199/) (2025)
- [Concept Drift Detection from Multi-Class Imbalanced Data Streams](https://mlsanity.com/paper/2104.10228/) (2021)
- [Embedding Graph Convolutional Networks in Recurrent Neural Networks for Predictive Monitoring](https://mlsanity.com/paper/2112.09641/) (2021)
- [Confidence intervals uncovered: Are we ready for real-world medical imaging AI?](https://mlsanity.com/paper/2409.17763/) (2024)
- [LLM Embeddings for Deep Learning on Tabular Data](https://mlsanity.com/paper/2502.11596/) (2025)
- [A Comparative Evaluation of Quantification Methods](https://mlsanity.com/paper/2103.03223/) (2021)
- [On the validity of pre-trained transformers for natural language processing in the software engineering domain](https://mlsanity.com/paper/2109.04738/) (2021)
- [Statistical Multicriteria Benchmarking via the GSD-Front](https://mlsanity.com/paper/2406.03924/) (2024)
- [Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Models](https://mlsanity.com/paper/2508.12461/) (2025)
- [Experimental Standards for Deep Learning in Natural Language Processing Research](https://mlsanity.com/paper/2204.06251/) (2022)
- [TASKED: Transformer-based Adversarial learning for human activity recognition using wearable sensors via Self-KnowledgE Distillation](https://mlsanity.com/paper/2209.09092/) (2022)
- [Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarks](https://mlsanity.com/paper/2501.04234/) (2025)
- [A Bayesian Bradley-Terry model to compare multiple ML algorithms on multiple data sets](https://mlsanity.com/paper/2208.04935/) (2022)
- [Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework](https://mlsanity.com/paper/2410.18653/) (2024)
- [Statistical Multicriteria Evaluation of LLM-Generated Text](https://mlsanity.com/paper/2506.18082/) (2025)
- [Semi-Supervised Constrained Clustering: An In-Depth Overview, Ranked Taxonomy and Future Research Directions](https://mlsanity.com/paper/2303.00522/) (2023)
- [Kernel Density Estimation for Multiclass Quantification](https://mlsanity.com/paper/2401.00490/) (2023)
- [Model-based metrics: Sample-efficient estimates of predictive model subpopulation performance](https://mlsanity.com/paper/2104.12231/) (2021)
- [Model Optimization in Imbalanced Regression](https://mlsanity.com/paper/2206.09991/) (2022)
- [When is an Embedding Model More Promising than Another?](https://mlsanity.com/paper/2406.07640/) (2024)
- [Towards a Data Privacy-Predictive Performance Trade-off](https://mlsanity.com/paper/2201.05226/) (2022)
- [Retweet communities reveal the main sources of hate speech](https://mlsanity.com/paper/2105.14898/) (2021)

## Abstract (arXiv)

The machine learning community adopted the use of null hypothesis significance testing (NHST) in order to ensure the statistical validity of results. Many scientific fields however realized the shortcomings of frequentist reasoning and in the most radical cases even banned its use in publications. We should do the same: just as we have embraced the Bayesian paradigm in the development of new machine learning methods, so we should also use it in the analysis of our own results. We argue for abandonment of NHST by exposing its fallacies and, more importantly, offer better - more sound and useful - alternatives for it.