# An Approach to Multiple Comparison Benchmark Evaluations that is Stable Under Manipulation of the Comparate Set

arXiv: https://arxiv.org/abs/2305.11921 · HTML: https://mlsanity.com/paper/2305.11921/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 45 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 25 of 25)

- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [InceptionTime: Finding AlexNet for Time Series Classification](https://mlsanity.com/paper/1909.04939/) (2019)
- [Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline](https://mlsanity.com/paper/1611.06455/) (2016)
- [ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernels](https://mlsanity.com/paper/1910.13051/) (2019)
- [MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classification](https://mlsanity.com/paper/2012.08791/) (2020)
- [HIVE-COTE 2.0: a new meta ensemble for time series classification](https://mlsanity.com/paper/2104.07551/) (2021)
- [Should we really use post-hoc tests based on mean-ranks?](https://mlsanity.com/paper/1505.02288/) (2015)
- [catch22: CAnonical Time-series CHaracteristics](https://mlsanity.com/paper/1901.10200/) (2019)
- [Bake off redux: a review and experimental evaluation of recent time series classification algorithms](https://mlsanity.com/paper/2304.13029/) (2023)
- [Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis](https://mlsanity.com/paper/1606.04316/) (2016)
- [MultiRocket: Multiple pooling operators and transformations for fast and effective time series classification](https://mlsanity.com/paper/2102.00457/) (2021)
- [A Time Series Forest for Classification and Feature Extraction](https://mlsanity.com/paper/1302.2277/) (2013)
- [TS-CHIEF: A Scalable and Accurate Forest Algorithm for Time Series Classification](https://mlsanity.com/paper/1906.10329/) (2019)
- [HYDRA: Competing convolutional kernels for fast and accurate time series classification](https://mlsanity.com/paper/2203.13652/) (2022)
- [The Canonical Interval Forest (CIF) Classifier for Time Series Classification](https://mlsanity.com/paper/2008.09172/) (2020)
- [Fast and Accurate Time Series Classification with WEASEL](https://mlsanity.com/paper/1701.07681/) (2017)
- [Proximity Forest: An effective and scalable distance-based classifier for time series](https://mlsanity.com/paper/1808.10594/) (2018)
- [The Temporal Dictionary Ensemble (TDE) Classifier for Time Series Classification](https://mlsanity.com/paper/2105.03841/) (2021)
- [Scalable Dictionary Classifiers for Time Series Classification](https://mlsanity.com/paper/1907.11815/) (2019)
- [Statistical Comparisons of Classifiers by Generalized Stochastic Dominance](https://mlsanity.com/paper/2209.01857/) (2022)
- [Statistical comparison of classifiers through Bayesian hierarchical modelling](https://mlsanity.com/paper/1609.08905/) (2016)
- [From BOP to BOSS and Beyond: Time Series Classification with Dictionary Based Classifiers](https://mlsanity.com/paper/1809.06751/) (2018)
- [A Bayesian Bradley-Terry model to compare multiple ML algorithms on multiple data sets](https://mlsanity.com/paper/2208.04935/) (2022)
- [Unsupervised Feature Based Algorithms for Time Series Extrinsic Regression](https://mlsanity.com/paper/2305.01429/) (2023)

## Built on by (most cited first, 29 of 29)

- [A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detection](https://mlsanity.com/paper/2307.03759/) (2023)
- [aeon: a Python toolkit for learning from time series](https://mlsanity.com/paper/2406.14231/) (2024)
- [QUANT: A Minimalist Interval Method for Time Series Classification](https://mlsanity.com/paper/2308.00928/) (2023)
- [Look Into the LITE in Deep Learning for Time Series Classification](https://mlsanity.com/paper/2409.02869/) (2024)
- [Fast, Interpretable, and Deterministic Time Series Classification With a Bag-of-Receptive-Fields](https://mlsanity.com/paper/2311.18029/) (2023)
- [Series2Vec: Similarity-based Self-supervised Representation Learning for Time Series Classification](https://mlsanity.com/paper/2312.03998/) (2023)
- [ShapeDBA: Generating Effective Time Series Prototypes using ShapeDTW Barycenter Averaging](https://mlsanity.com/paper/2309.16353/) (2023)
- [MONSTER: Monash Scalable Time Series Evaluation Repository](https://mlsanity.com/paper/2502.15122/) (2025)
- [Finding Foundation Models for Time Series Classification with a PreText Task](https://mlsanity.com/paper/2311.14534/) (2023)
- [Early Classification of Time Series: A Survey and Benchmark](https://mlsanity.com/paper/2406.18332/) (2024)
- [The Multiverse of Time Series Machine Learning: an Archive for Multivariate Time Series Classification](https://mlsanity.com/paper/2603.20352/) (2026)
- [Time series classification with random convolution kernels: pooling operators and input representations matter](https://mlsanity.com/paper/2409.01115/) (2024)
- [Rock the KASBA: Blazingly Fast and Accurate Time Series Clustering](https://mlsanity.com/paper/2411.17838/) (2024)
- [A Standardized Benchmark for Skeleton-Based Rehabilitation Assessment Using Deep Learning](https://mlsanity.com/paper/2507.21018/) (2025)
- [Convolutional and Deep Learning based techniques for Time Series Ordinal Classification](https://mlsanity.com/paper/2306.10084/) (2023)
- [Watch Your Step: A Cost-Sensitive Framework for Accelerometer-Based Fall Detection in Real-World Streaming Scenarios](https://mlsanity.com/paper/2509.11789/) (2025)
- [Learning Time-Series Representations by Hierarchical Uniformity-Tolerance Latent Balancing](https://mlsanity.com/paper/2510.01658/) (2025)
- [Enhancing Time Series Classification with Diversity-Driven Neural Network Ensembles](https://mlsanity.com/paper/2602.07579/) (2026)
- [A Wave is Worth 100 Words: Investigating Cross-Domain Transferability in Time Series](https://mlsanity.com/paper/2412.00772/) (2024)
- [Semi-Periodic Activation for Time Series Classification](https://mlsanity.com/paper/2412.09889/) (2024)
- [Anomalous Agreement: How to find the Ideal Number of Anomaly Classes in Correlated, Multivariate Time Series Data](https://mlsanity.com/paper/2501.07172/) (2025)
- [PRISM: Lightweight Multivariate Time-Series Classification through Symmetric Multi-Resolution Convolutional Layers](https://mlsanity.com/paper/2508.04503/) (2025)
- [Adaptive Structured Pruning of Convolutional Neural Networks for Time Series Classification](https://mlsanity.com/paper/2602.12744/) (2026)
- [Soft-MSM: Differentiable Context-Aware Elastic Alignment for Time Series](https://mlsanity.com/paper/2605.00069/) (2026)
- [Scaling Time Series Classification via XAI-Driven Data Reduction](https://mlsanity.com/paper/2607.15774/) (2026)
- [Statistical comparisons of time-series feature sets on classification tasks](https://mlsanity.com/paper/2608.01586/) (2026)
- [Improving Multivariate Time Series Classification with Class-Wise Training and Model Aggregation](https://mlsanity.com/paper/2609.07493/) (2026)
- [CLUES-WEASEL: No additional clues required to choose your time series clustering algorithm](https://mlsanity.com/paper/2609.07606/) (2026)
- [Adapting prior-data fitted networks for tabular anomaly detection](https://mlsanity.com/paper/2610.06693/) (2026)

## Abstract (arXiv)

The measurement of progress using benchmarks evaluations is ubiquitous in computer science and machine learning. However, common approaches to analyzing and presenting the results of benchmark comparisons of multiple algorithms over multiple datasets, such as the critical difference diagram introduced by Demvsar (2006), have important shortcomings and, we show, are open to both inadvertent and intentional manipulation. To address these issues, we propose a new approach to presenting the results of benchmark comparisons, the Multiple Comparison Matrix (MCM), that prioritizes pairwise comparisons and precludes the means of manipulating experimental results in existing approaches. MCM can be used to show the results of an all-pairs comparison, or to show the results of a comparison between one or more selected algorithms and the state of the art. MCM is implemented in Python and is publicly available.