# Better than classical? The subtle art of benchmarking quantum machine learning models

arXiv: https://arxiv.org/abs/2403.07059 · HTML: https://mlsanity.com/paper/2403.07059/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 103 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 30 of 37)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Supervised learning with quantum enhanced feature spaces](https://mlsanity.com/paper/1804.11326/) (2018)
- [Barren plateaus in quantum neural network training landscapes](https://mlsanity.com/paper/1803.11173/) (2018)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Power of data in quantum machine learning](https://mlsanity.com/paper/2011.01938/) (2020)
- [The effect of data encoding on the expressive power of variational quantum machine learning models](https://mlsanity.com/paper/2008.08605/) (2020)
- [The Intrinsic Dimension of Images and Its Impact on Learning](https://mlsanity.com/paper/2104.08894/) (2021)
- [Optimizing Millions of Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/1911.02590/) (2019)
- [Are GANs Created Equal? A Large-Scale Study](https://mlsanity.com/paper/1711.10337/) (2017)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [A General Theory of Equivariant CNNs on Homogeneous Spaces](https://mlsanity.com/paper/1811.02017/) (2018)
- [Do CIFAR-10 Classifiers Generalize to CIFAR-10?](https://mlsanity.com/paper/1806.00451/) (2018)
- [Supervised quantum machine learning models are kernel methods](https://mlsanity.com/paper/2101.11020/) (2021)
- [Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling](https://mlsanity.com/paper/1802.09127/) (2018)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [Transfer learning in hybrid classical-quantum neural networks](https://mlsanity.com/paper/1912.08278/) (2019)
- [Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research](https://mlsanity.com/paper/2112.01716/) (2021)
- [The Inductive Bias of Quantum Kernels](https://mlsanity.com/paper/2106.03747/) (2021)
- [Do Transformer Modifications Transfer Across Implementations and Applications?](https://mlsanity.com/paper/2102.11972/) (2021)
- [Recurrent Quantum Neural Networks](https://mlsanity.com/paper/2006.14619/) (2020)
- [Group-Invariant Quantum Machine Learning](https://mlsanity.com/paper/2205.02261/) (2022)
- [Classical surrogates for quantum learning models](https://mlsanity.com/paper/2206.11740/) (2022)
- [Deep Networks and the Multiple Manifold Problem](https://mlsanity.com/paper/2008.11245/) (2020)
- [On quantum backpropagation, information reuse, and cheating measurement collapse](https://mlsanity.com/paper/2305.13362/) (2023)
- [Entangled Datasets for Quantum Machine Learning](https://mlsanity.com/paper/2109.03400/) (2021)
- [Value-laden Disciplinary Shifts in Machine Learning](https://mlsanity.com/paper/1912.01172/) (2019)

## Built on by (most cited first, 30 of 85)

- [Quantum Architecture Search: A Survey](https://mlsanity.com/paper/2406.06210/) (2024)
- [Quantum Kernel Methods under Scrutiny: A Benchmarking Study](https://mlsanity.com/paper/2409.04406/) (2024)
- [A Primer on Quantum Machine Learning](https://mlsanity.com/paper/2511.15969/) (2025)
- [Benchmarking quantum machine learning kernel training for classification tasks](https://mlsanity.com/paper/2408.10274/) (2024)
- [Benchmarking Quantum Reinforcement Learning](https://mlsanity.com/paper/2501.15893/) (2025)
- [Reinforcement learning-based architecture search for quantum machine learning](https://mlsanity.com/paper/2406.02717/) (2024)
- [Opportunities and limitations of explaining quantum machine learning](https://mlsanity.com/paper/2412.14753/) (2024)
- [Quantum Neural Networks in Practice: A Comparative Study with Classical Models from Standard Data Sets to Industrial Images](https://mlsanity.com/paper/2411.19276/) (2024)
- [Data re-uploading in Quantum Machine Learning for time series: application to traffic forecasting](https://mlsanity.com/paper/2501.12776/) (2025)
- [Quantum-Audit: Evaluating the Reasoning Limits of LLMs on Quantum Computing](https://mlsanity.com/paper/2602.10092/) (2026)
- [Guided-SPSA: Simultaneous Perturbation Stochastic Approximation assisted by the Parameter Shift Rule](https://mlsanity.com/paper/2404.15751/) (2024)
- [QUACK: Quantum Aligned Centroid Kernel](https://mlsanity.com/paper/2405.00304/) (2024)
- [Quantum Hamiltonian Embedding of Images for Data Reuploading Classifiers](https://mlsanity.com/paper/2407.14055/) (2024)
- [Leveraging Diffusion Models for Parameterized Quantum Circuit Generation](https://mlsanity.com/paper/2505.20863/) (2025)
- [Spectral methods: crucial for machine learning, natural for quantum computers?](https://mlsanity.com/paper/2603.24654/) (2026)
- [Exploiting the equivalence between quantum neural networks and perceptrons](https://mlsanity.com/paper/2407.04371/) (2024)
- [Stability and Generalization of Quantum Neural Networks](https://mlsanity.com/paper/2501.12737/) (2025)
- [On the similarity of bandwidth-tuned quantum kernels and classical kernels](https://mlsanity.com/paper/2503.05602/) (2025)
- [AI Benchmark Democratization and Carpentry](https://mlsanity.com/paper/2512.11588/) (2025)
- [MerLin: A Discovery Engine for Photonic and Hybrid Quantum Machine Learning](https://mlsanity.com/paper/2602.11092/) (2026)
- [Security Concerns in Quantum Machine Learning as a Service](https://mlsanity.com/paper/2408.09562/) (2024)
- [The Impact of Feature Embedding Placement in the Ansatz of a Quantum Kernel in QSVMs](https://mlsanity.com/paper/2409.13147/) (2024)
- [Satellite image classification with neural quantum kernels](https://mlsanity.com/paper/2409.20356/) (2024)
- [Regression and Classification with Single-Qubit Quantum Neural Networks](https://mlsanity.com/paper/2412.09486/) (2024)
- [Multiple Embeddings for Quantum Machine Learning](https://mlsanity.com/paper/2503.22758/) (2025)
- [TabularQGAN: A quantum generative model for tabular data synthesis](https://mlsanity.com/paper/2505.22533/) (2025)
- [An MLCommons Scientific Benchmarks Ontology](https://mlsanity.com/paper/2511.05614/) (2025)
- [Learning Reduced Representations for Quantum Classifiers](https://mlsanity.com/paper/2512.01509/) (2025)
- [Hybrid Classical-Quantum Transfer Learning with Noisy Quantum Circuits](https://mlsanity.com/paper/2603.16973/) (2026)
- [Non-variational supervised quantum kernel methods: a review](https://mlsanity.com/paper/2604.07896/) (2026)

## Abstract (arXiv)

Benchmarking models via classical simulations is one of the main ways to judge ideas in quantum machine learning before noise-free hardware is available. However, the huge impact of the experimental design on the results, the small scales within reach today, as well as narratives influenced by the commercialisation of quantum technologies make it difficult to gain robust insights. To facilitate better decision-making we develop an open-source package based on the PennyLane software framework and use it to conduct a large-scale study that systematically tests 12 popular quantum machine learning models on 6 binary classification tasks used to create 160 individual datasets. We find that overall, out-of-the-box classical machine learning models outperform the quantum classifiers. Moreover, removing entanglement from a quantum model often results in as good or better performance, suggesting that "quantumness" may not be the crucial ingredient for the small learning tasks considered here. Our benchmarks also unlock investigations beyond simplistic leaderboard comparisons, and we identify five important questions for quantum model design that follow from our results.