# Benchmarking Automatic Machine Learning Frameworks

arXiv: https://arxiv.org/abs/1808.06492 · HTML: https://mlsanity.com/paper/1808.06492/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Aug 2018.

## Built on by (most cited first, 15 of 15)

- [AMLB: an AutoML Benchmark](https://mlsanity.com/paper/2207.12560/) (2022)
- [Towards Green Automated Machine Learning: Status Quo and Future Directions](https://mlsanity.com/paper/2111.05850/) (2021)
- [Task Selection for AutoML System Evaluation](https://mlsanity.com/paper/2208.12754/) (2022)
- [Automated machine learning: AI-driven decision making in business analytics](https://mlsanity.com/paper/2205.10538/) (2022)
- [The Potential of AutoML for Recommender Systems](https://mlsanity.com/paper/2402.04453/) (2024)
- [A Neophyte With AutoML: Evaluating the Promises of Automatic Machine Learning Tools](https://mlsanity.com/paper/2101.05840/) (2021)
- [Assessing the Use of AutoML for Data-Driven Software Engineering](https://mlsanity.com/paper/2307.10774/) (2023)
- [Benchmarking AutoML Frameworks for Disease Prediction Using Medical Claims](https://mlsanity.com/paper/2107.10495/) (2021)
- [AutoML Systems For Medical Imaging](https://mlsanity.com/paper/2306.04750/) (2023)
- [Post-processing Multi-Model Medium-Term Precipitation Forecasts Using Convolutional Neural Networks](https://mlsanity.com/paper/2105.07043/) (2021)
- [Surrogate Model Based Hyperparameter Tuning for Deep Learning with SPOT](https://mlsanity.com/paper/2105.14625/) (2021)
- [Predictability of Machine Learning Algorithms and Related Feature Extraction Techniques](https://mlsanity.com/paper/2305.00449/) (2023)
- [STREAMLINE: An Automated Machine Learning Pipeline for Biomedicine Applied to Examine the Utility of Photography-Based Phenotypes for OSA Prediction Across International Sleep Centers](https://mlsanity.com/paper/2312.05461/) (2023)
- [BOASF: A Unified Framework for Speeding up Automatic Machine Learning via Adaptive Successive Filtering](https://mlsanity.com/paper/2507.20446/) (2025)
- [AutoML-Med: A Framework for Automated Machine Learning in Medical Tabular Data](https://mlsanity.com/paper/2508.02625/) (2025)

## Abstract (arXiv)

AutoML serves as the bridge between varying levels of expertise when designing machine learning systems and expedites the data science process. A wide range of techniques is taken to address this, however there does not exist an objective comparison of these techniques. We present a benchmark of current open source AutoML solutions using open source datasets. We test auto-sklearn, TPOT, auto_ml, and H2O's AutoML solution against a compiled set of regression and classification datasets sourced from OpenML and find that auto-sklearn performs the best across classification datasets and TPOT performs the best across regression datasets.