# Generative and reproducible benchmarks for comprehensive evaluation of machine learning classifiers

arXiv: https://arxiv.org/abs/2107.06475 · HTML: https://mlsanity.com/paper/2107.06475/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 32 references that are in the map.
- First posted on arXiv 14 Jul 2021.

## Builds on (most cited first, 6 of 6)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning](https://mlsanity.com/paper/2007.04074/) (2020)
- [PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparison](https://mlsanity.com/paper/1703.00512/) (2017)
- [Where are we now? A large benchmark study of recent symbolic regression methods](https://mlsanity.com/paper/1804.09331/) (2018)

## Built on by (most cited first, 4 of 4)

- [Benchmarking AutoML algorithms on a collection of synthetic classification problems](https://mlsanity.com/paper/2212.02704/) (2022)
- [Faster Convergence with Lexicase Selection in Tree-based Automated Machine Learning](https://mlsanity.com/paper/2302.00731/) (2023)
- [Multi-Dimensional Ability Diagnosis for Machine Learning Algorithms](https://mlsanity.com/paper/2307.07134/) (2023)
- [Generating Benchmark Health Data Using a Tabular Diffusion Transformer](https://mlsanity.com/paper/2608.14496/) (2026)

## Abstract (arXiv)

Understanding the strengths and weaknesses of machine learning (ML) algorithms is crucial for determine their scope of application. Here, we introduce the DIverse and GENerative ML Benchmark (DIGEN) - a collection of synthetic datasets for comprehensive, reproducible, and interpretable benchmarking of machine learning algorithms for classification of binary outcomes. The DIGEN resource consists of 40 mathematical functions which map continuous features to discrete endpoints for creating synthetic datasets. These 40 functions were discovered using a heuristic algorithm designed to maximize the diversity of performance among multiple popular machine learning algorithms thus providing a useful test suite for evaluating and comparing new methods. Access to the generative functions facilitates understanding of why a method performs poorly compared to other algorithms thus providing ideas for improvement. The resource with extensive documentation and analyses is open-source and available on GitHub.