# PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparison

arXiv: https://arxiv.org/abs/1703.00512 · HTML: https://mlsanity.com/paper/1703.00512/

## Key facts

- Cited by 140 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2017.

## Built on by (most cited first, 30 of 140)

- [PDEBENCH: An Extensive Benchmark for Scientific Machine Learning](https://mlsanity.com/paper/2210.07182/) (2022)
- [Interpretable Scientific Discovery with Symbolic Regression: A Review](https://mlsanity.com/paper/2211.10873/) (2022)
- [Transformer-based Planning for Symbolic Regression](https://mlsanity.com/paper/2303.06833/) (2023)
- [NICE: An Algorithm for Nearest Instance Counterfactual Explanations](https://mlsanity.com/paper/2104.07411/) (2021)
- [On Explaining Random Forests with SAT](https://mlsanity.com/paper/2105.10278/) (2021)
- [AMLB: an AutoML Benchmark](https://mlsanity.com/paper/2207.12560/) (2022)
- [SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training](https://mlsanity.com/paper/2310.02227/) (2023)
- [On Efficiently Explaining Graph-Based Classifiers](https://mlsanity.com/paper/2106.01350/) (2021)
- [The Inadequacy of Shapley Values for Explainability](https://mlsanity.com/paper/2302.08160/) (2023)
- [Exhaustive Symbolic Regression](https://mlsanity.com/paper/2211.11461/) (2022)
- [Model scale versus domain knowledge in statistical forecasting of chaotic systems](https://mlsanity.com/paper/2303.08011/) (2023)
- [AlphaD3M: Machine Learning Pipeline Synthesis](https://mlsanity.com/paper/2111.02508/) (2021)
- [Eliminating The Impossible, Whatever Remains Must Be True](https://mlsanity.com/paper/2206.09551/) (2022)
- [Feature Necessity & Relevancy in ML Classifier Explanations](https://mlsanity.com/paper/2210.15675/) (2022)
- [Controllable Neural Symbolic Regression](https://mlsanity.com/paper/2304.10336/) (2023)
- [Taylor Genetic Programming for Symbolic Regression](https://mlsanity.com/paper/2205.09751/) (2022)
- [On Computing Probabilistic Abductive Explanations](https://mlsanity.com/paper/2212.05990/) (2022)
- [Symbolic regression via MDLformer-guided search: from minimizing prediction error to minimizing description length](https://mlsanity.com/paper/2411.03753/) (2024)
- [Call for Action: towards the next generation of symbolic regression benchmark](https://mlsanity.com/paper/2505.03977/) (2025)
- [Delivering Inflated Explanations](https://mlsanity.com/paper/2306.15272/) (2023)
- [A Neural-Guided Dynamic Symbolic Network for Exploring Mathematical Expressions from Data](https://mlsanity.com/paper/2309.13705/) (2023)
- [Think like a Scientist: Physics-guided LLM Agent for Equation Discovery](https://mlsanity.com/paper/2602.12259/) (2026)
- [Efficient Explanations With Relevant Sets](https://mlsanity.com/paper/2106.00546/) (2021)
- [Measuring the Energy Consumption and Efficiency of Deep Neural Networks: An Empirical Analysis and Design Recommendations](https://mlsanity.com/paper/2403.08151/) (2024)
- [On Formal Feature Attribution and Its Approximation](https://mlsanity.com/paper/2307.03380/) (2023)
- [Investigating the Histogram Loss in Regression](https://mlsanity.com/paper/2402.13425/) (2024)
- [Statistical Multicriteria Benchmarking via the GSD-Front](https://mlsanity.com/paper/2406.03924/) (2024)
- [Efficient Explanations for Knowledge Compilation Languages](https://mlsanity.com/paper/2107.01654/) (2021)
- [Benchmark Data Repositories for Better Benchmarking](https://mlsanity.com/paper/2410.24100/) (2024)
- [ControlBurn: Feature Selection by Sparse Forests](https://mlsanity.com/paper/2107.00219/) (2021)

## Abstract (arXiv)

The selection, development, or comparison of machine learning methods in data mining can be a difficult task based on the target problem and goals of a particular study. Numerous publicly available real-world and simulated benchmark datasets have emerged from different sources, but their organization and adoption as standards have been inconsistent. As such, selecting and curating specific benchmarks remains an unnecessary burden on machine learning practitioners and data scientists. The present study introduces an accessible, curated, and developing public benchmark resource to facilitate identification of the strengths and weaknesses of different machine learning methodologies. We compare meta-features among the current set of benchmark datasets in this resource to characterize the diversity of available data. Finally, we apply a number of established machine learning methods to the entire benchmark suite and analyze how datasets and algorithms cluster in terms of performance. This work is an important first step towards understanding the limitations of popular benchmarking suites and developing a resource that connects existing benchmarking standards to more diverse and efficient standards in the future.