# PMLBmini: A Tabular Classification Benchmark Suite for Data-Scarce Applications

arXiv: https://arxiv.org/abs/2409.01635 · HTML: https://mlsanity.com/paper/2409.01635/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 53 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning](https://mlsanity.com/paper/2007.04074/) (2020)
- [PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparison](https://mlsanity.com/paper/1703.00512/) (2017)
- [Should we really use post-hoc tests based on mean-ranks?](https://mlsanity.com/paper/1505.02288/) (2015)
- [An Open Source AutoML Benchmark](https://mlsanity.com/paper/1907.00909/) (2019)
- [How Complex is your classification problem? A survey on measuring classification complexity](https://mlsanity.com/paper/1808.03591/) (2018)
- [PMLB v1.0: An open source dataset collection for benchmarking machine learning methods](https://mlsanity.com/paper/2012.00058/) (2020)
- [AMLB: an AutoML Benchmark](https://mlsanity.com/paper/2207.12560/) (2022)
- [JuMP: A Modeling Language for Mathematical Optimization](https://mlsanity.com/paper/1508.01982/) (2015)
- [GitTables: A Large-Scale Corpus of Relational Tables](https://mlsanity.com/paper/2106.07258/) (2021)
- [TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications](https://mlsanity.com/paper/2311.02971/) (2023)
- [AutoPrognosis: Automated Clinical Prognostic Modeling via Bayesian Optimization with Structured Kernel Learning](https://mlsanity.com/paper/1802.07207/) (2018)
- [HyperFast: Instant Classification for Tabular Data](https://mlsanity.com/paper/2402.14335/) (2024)
- [Scaling TabPFN: Sketching and Feature Selection for Tabular Prior-Data Fitted Networks](https://mlsanity.com/paper/2311.10609/) (2023)
- [AutoPrognosis 2.0: Democratizing Diagnostic and Prognostic Modeling in Healthcare with Automated Machine Learning](https://mlsanity.com/paper/2210.12090/) (2022)
- [Safe Screening for Logistic Regression with ℓ0-ℓ2 Regularization](https://mlsanity.com/paper/2202.00467/) (2022)
- [To tune or not to tune, a case study of ridge logistic regression in small or sparse datasets](https://mlsanity.com/paper/2101.11230/) (2021)
- [Cost-Sensitive Best Subset Selection for Logistic Regression: A Mixed-Integer Conic Optimization Perspective](https://mlsanity.com/paper/2310.05464/) (2023)

## Built on by (most cited first, 6 of 6)

- [Early Stopping Tabular In-Context Learning](https://mlsanity.com/paper/2506.21387/) (2025)
- [Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models](https://mlsanity.com/paper/2605.06510/) (2026)
- ["Oh LLM, I'm Asking Thee, Please Give Me a Decision Tree": Zero-Shot Decision Tree Induction and Embedding with Large Language Models](https://mlsanity.com/paper/2409.18594/) (2024)
- [RamanBench: A Large-Scale Benchmark for Machine Learning on Raman Spectroscopy](https://mlsanity.com/paper/2605.02003/) (2026)
- [Recovering Wasted Compute in Autoresearch Agents](https://mlsanity.com/paper/2608.10424/) (2026)
- [Explaining Tabular Foundation Model Differences Through Meta-Features](https://mlsanity.com/paper/2605.28418/) (2026)

## Abstract (arXiv)

In practice, we are often faced with small-sized tabular data. However, current tabular benchmarks are not geared towards data-scarce applications, making it very difficult to derive meaningful conclusions from empirical comparisons. We introduce PMLBmini, a tabular benchmark suite of 44 binary classification datasets with sample sizes ≤ 500. We use our suite to thoroughly evaluate current automated machine learning (AutoML) frameworks, off-the-shelf tabular deep neural networks, as well as classical linear models in the low-data regime. Our analysis reveals that state-of-the-art AutoML and deep learning approaches often fail to appreciably outperform even a simple logistic regression baseline, but we also identify scenarios where AutoML and deep learning methods are indeed reasonable to apply. Our benchmark suite, available on https://github.com/RicardoKnauer/TabMini , allows researchers and practitioners to analyze their own methods and challenge their data efficiency.