# Should we really use post-hoc tests based on mean-ranks?

arXiv: https://arxiv.org/abs/1505.02288 · HTML: https://mlsanity.com/paper/1505.02288/

## Key facts

- Cited by 116 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 May 2015.

## Built on by (most cited first, 30 of 116)

- [HIVE-COTE 2.0: a new meta ensemble for time series classification](https://mlsanity.com/paper/2104.07551/) (2021)
- [Bake off redux: a review and experimental evaluation of recent time series classification algorithms](https://mlsanity.com/paper/2304.13029/) (2023)
- [HYDRA: Competing convolutional kernels for fast and accurate time series classification](https://mlsanity.com/paper/2203.13652/) (2022)
- [How I failed machine learning in medical imaging -- shortcomings and recommendations](https://mlsanity.com/paper/2103.10292/) (2021)
- [Multi-Label Classification Neural Networks with Hard Logical Constraints](https://mlsanity.com/paper/2103.13427/) (2021)
- [An Approach to Multiple Comparison Benchmark Evaluations that is Stable Under Manipulation of the Comparate Set](https://mlsanity.com/paper/2305.11921/) (2023)
- [A Large-Scale Study of Probabilistic Calibration in Neural Network Regression](https://mlsanity.com/paper/2306.02738/) (2023)
- [The Temporal Dictionary Ensemble (TDE) Classifier for Time Series Classification](https://mlsanity.com/paper/2105.03841/) (2021)
- [Statistical Comparisons of Classifiers by Generalized Stochastic Dominance](https://mlsanity.com/paper/2209.01857/) (2022)
- [The FreshPRINCE: A Simple Transformation Based Pipeline Time Series Classifier](https://mlsanity.com/paper/2201.12048/) (2022)
- [TodyNet: Temporal Dynamic Graph Neural Network for Multivariate Time Series Classification](https://mlsanity.com/paper/2304.05078/) (2023)
- [AQuA: A Benchmarking Tool for Label Quality Assessment](https://mlsanity.com/paper/2306.09467/) (2023)
- [Vote'n'Rank: Revision of Benchmarking with Social Choice Theory](https://mlsanity.com/paper/2210.05769/) (2022)
- [ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities](https://mlsanity.com/paper/2412.06745/) (2024)
- [A Review and Evaluation of Elastic Distance Functions for Time Series Clustering](https://mlsanity.com/paper/2205.15181/) (2022)
- [Episodic Memories Generation and Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2501.13121/) (2025)
- [Learnable Dynamic Temporal Pooling for Time Series Classification](https://mlsanity.com/paper/2104.02577/) (2021)
- [From Variability to Stability: Advancing RecSys Benchmarking Practices](https://mlsanity.com/paper/2402.09766/) (2024)
- [Reproducibility in Multiple Instance Learning: A Case For Algorithmic Unit Tests](https://mlsanity.com/paper/2310.17867/) (2023)
- [Statistical Multicriteria Benchmarking via the GSD-Front](https://mlsanity.com/paper/2406.03924/) (2024)
- [A Bayesian Bradley-Terry model to compare multiple ML algorithms on multiple data sets](https://mlsanity.com/paper/2208.04935/) (2022)
- [Privacy Meets Explainability: A Comprehensive Impact Benchmark](https://mlsanity.com/paper/2211.04110/) (2022)
- [Unsupervised Feature Based Algorithms for Time Series Extrinsic Regression](https://mlsanity.com/paper/2305.01429/) (2023)
- [Look Into the LITE in Deep Learning for Time Series Classification](https://mlsanity.com/paper/2409.02869/) (2024)
- [Elastic Similarity and Distance Measures for Multivariate Time Series](https://mlsanity.com/paper/2102.10231/) (2021)
- [A Siren Song of Open Source Reproducibility](https://mlsanity.com/paper/2204.04372/) (2022)
- [Scalable Classifier-Agnostic Channel Selection for Multivariate Time Series Classification](https://mlsanity.com/paper/2206.09274/) (2022)
- [Kernel Density Estimation for Multiclass Quantification](https://mlsanity.com/paper/2401.00490/) (2023)
- [Average Localised Proximity: A new data descriptor with good default one-class classification performance](https://mlsanity.com/paper/2101.11037/) (2021)
- [MrSQM: Fast Time Series Classification with Symbolic Representations](https://mlsanity.com/paper/2109.01036/) (2021)

## Abstract (arXiv)

The statistical comparison of multiple algorithms over multiple data sets is fundamental in machine learning. This is typically carried out by the Friedman test. When the Friedman test rejects the null hypothesis, multiple comparisons are carried out to establish which are the significant differences among algorithms. The multiple comparisons are usually performed using the mean-ranks test. The aim of this technical note is to discuss the inconsistencies of the mean-ranks post-hoc test with the goal of discouraging its use in machine learning as well as in medicine, psychology, etc.. We show that the outcome of the mean-ranks test depends on the pool of algorithms originally included in the experiment. In other words, the outcome of the comparison between algorithms A and B depends also on the performance of the other algorithms included in the original experiment. This can lead to paradoxical situations. For instance the difference between A and B could be declared significant if the pool comprises algorithms C, D, E and not significant if the pool comprises algorithms F, G, H. To overcome these issues, we suggest instead to perform the multiple comparison using a test whose outcome only depends on the two algorithms being compared, such as the sign-test or the Wilcoxon signed-rank test.