mlsanityData to 7 Oct 2026

An Approach to Multiple Comparison Benchmark Evaluations that is Stable Under Manipulation of the Comparate Set

arXiv 2305.11921 ↗First posted 19 May 2023stat.ME cs.AI cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

The measurement of progress using benchmarks evaluations is ubiquitous in computer science and machine learning. However, common approaches to analyzing and presenting the results of benchmark comparisons of multiple algorithms over multiple datasets, such as the critical difference diagram introduced by Demvsar (2006), have important shortcomings and, we show, are open to both inadvertent and intentional manipulation. To address these issues, we propose a new approach to presenting the results of benchmark comparisons, the Multiple Comparison Matrix (MCM), that prioritizes pairwise comparisons and precludes the means of manipulating experimental results in existing approaches. MCM can be used to show the results of an all-pairs comparison, or to show the results of a comparison between one or more selected algorithms and the state of the art. MCM is implemented in Python and is publicly available.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 25 of its 45 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

25 of 45 references.

References · 25

  1. 2018Deep learning for time series classification: a reviewcited by 575
  2. 2018The UCR Time Series Archivecited by 421
  3. 2019InceptionTime: Finding AlexNet for Time Series Classificationcited by 386
  4. 2016Time Series Classification from Scratch with Deep Neural Networks: A Strong Baselinecited by 358
  5. 2019ROCKET: Exceptionally fast and accurate time series classification using random convolutional kernelscited by 336
  6. 2020MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classificationcited by 186
  7. 2021HIVE-COTE 2.0: a new meta ensemble for time series classificationcited by 118
  8. 2015Should we really use post-hoc tests based on mean-ranks?cited by 116
  9. 2019catch22: CAnonical Time-series CHaracteristicscited by 103
  10. 2023Bake off redux: a review and experimental evaluation of recent time series classification algorithmscited by 99
  11. 2016Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysiscited by 92
  12. 2021MultiRocket: Multiple pooling operators and transformations for fast and effective time series classificationcited by 89
13 more
  1. 2013A Time Series Forest for Classification and Feature Extractioncited by 69
  2. 2019TS-CHIEF: A Scalable and Accurate Forest Algorithm for Time Series Classificationcited by 67
  3. 2022HYDRA: Competing convolutional kernels for fast and accurate time series classificationcited by 53
  4. 2020The Canonical Interval Forest (CIF) Classifier for Time Series Classificationcited by 48
  5. 2017Fast and Accurate Time Series Classification with WEASELcited by 47
  6. 2018Proximity Forest: An effective and scalable distance-based classifier for time seriescited by 46
  7. 2021The Temporal Dictionary Ensemble (TDE) Classifier for Time Series Classificationcited by 26
  8. 2019Scalable Dictionary Classifiers for Time Series Classificationcited by 20
  9. 2022Statistical Comparisons of Classifiers by Generalized Stochastic Dominancecited by 20
  10. 2018From BOP to BOSS and Beyond: Time Series Classification with Dictionary Based Classifierscited by 14
  11. 2016Statistical comparison of classifiers through Bayesian hierarchical modellingcited by 13
  12. 2022A Bayesian Bradley-Terry model to compare multiple ML algorithms on multiple data setscited by 9
  13. 2023Unsupervised Feature Based Algorithms for Time Series Extrinsic Regressioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 7; 2025: 6; 2026: 7.

27 papers.

Sort:

  1. 2023A Survey on Graph Neural Networks for Time Series: Forecasting, Classification, Imputation, and Anomaly Detectioncited by 145
  2. 2024aeon: a Python toolkit for learning from time seriescited by 46
  3. 2023QUANT: A Minimalist Interval Method for Time Series Classificationcited by 15
  4. 2024Look Into the LITE in Deep Learning for Time Series Classificationcited by 9
  5. 2023Fast, Interpretable, and Deterministic Time Series Classification With a Bag-of-Receptive-Fieldscited by 8
  6. 2023Series2Vec: Similarity-based Self-supervised Representation Learning for Time Series Classificationcited by 8
  7. 2023ShapeDBA: Generating Effective Time Series Prototypes using ShapeDTW Barycenter Averagingcited by 6
  8. 2025MONSTER: Monash Scalable Time Series Evaluation Repositorycited by 6
  9. 2023Finding Foundation Models for Time Series Classification with a PreText Taskcited by 4
  10. 2024Early Classification of Time Series: A Survey and Benchmarkcited by 3
  11. 2026The Multiverse of Time Series Machine Learning: an Archive for Multivariate Time Series Classificationcited by 3
  12. 2024Time series classification with random convolution kernels: pooling operators and input representations mattercited by 2
15 more
  1. 2025A Standardized Benchmark for Skeleton-Based Rehabilitation Assessment Using Deep Learningcited by 2
  2. 2023Convolutional and Deep Learning based techniques for Time Series Ordinal Classificationcited by 1
  3. 2024Rock the KASBA: Blazingly Fast and Accurate Time Series Clusteringcited by 1
  4. 2025Watch Your Step: A Cost-Sensitive Framework for Accelerometer-Based Fall Detection in Real-World Streaming Scenarioscited by 1
  5. 2025Learning Time-Series Representations by Hierarchical Uniformity-Tolerance Latent Balancingcited by 1
  6. 2026Enhancing Time Series Classification with Diversity-Driven Neural Network Ensemblescited by 1
  7. 2024A Wave is Worth 100 Words: Investigating Cross-Domain Transferability in Time Seriescited by 0
  8. 2024Semi-Periodic Activation for Time Series Classificationcited by 0
  9. 2025Anomalous Agreement: How to find the Ideal Number of Anomaly Classes in Correlated, Multivariate Time Series Datacited by 0
  10. 2025PRISM: Lightweight Multivariate Time-Series Classification through Symmetric Multi-Resolution Convolutional Layerscited by 0
  11. 2026Adaptive Structured Pruning of Convolutional Neural Networks for Time Series Classificationcited by 0
  12. 2026Soft-MSM: Differentiable Context-Aware Elastic Alignment for Time Seriescited by 0
  13. 2026Scaling Time Series Classification via XAI-Driven Data Reductioncited by 0
  14. 2026Statistical comparisons of time-series feature sets on classification taskscited by 0
  15. 2026Improving Multivariate Time Series Classification with Class-Wise Training and Model Aggregationcited by 0