# AutoMLBench: A Comprehensive Experimental Evaluation of Automated Machine Learning Frameworks

arXiv: https://arxiv.org/abs/2204.08358 · HTML: https://mlsanity.com/paper/2204.08358/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 52 references that are in the map.
- First posted on arXiv 18 Apr 2022.

## Builds on (most cited first, 14 of 14)

- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [Meta-Learning: A Survey](https://mlsanity.com/paper/1810.03548/) (2018)
- [Auto-Keras: An Efficient Neural Architecture Search System](https://mlsanity.com/paper/1806.10282/) (2018)
- [FLAML: A Fast and Lightweight AutoML Library](https://mlsanity.com/paper/1911.04706/) (2019)
- [An Open Source AutoML Benchmark](https://mlsanity.com/paper/1907.00909/) (2019)
- [Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science](https://mlsanity.com/paper/1603.06212/) (2016)
- [LightAutoML: AutoML Solution for a Large Financial Services Ecosystem](https://mlsanity.com/paper/2109.01528/) (2021)
- [Automated Machine Learning: State-of-The-Art and Open Challenges](https://mlsanity.com/paper/1906.02287/) (2019)
- [Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Tools](https://mlsanity.com/paper/1908.05557/) (2019)
- [GAMA: a General Automated Machine learning Assistant](https://mlsanity.com/paper/2007.04911/) (2020)
- [The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Development](https://mlsanity.com/paper/1905.08942/) (2019)

## Built on by (most cited first, 3 of 3)

- [What fifty-one years of Linguistics and Artificial Intelligence research tell us about their correlation: A scientometric analysis](https://mlsanity.com/paper/2411.19858/) (2024)
- [GenoTEX: An LLM Agent Benchmark for Automated Gene Expression Data Analysis](https://mlsanity.com/paper/2406.15341/) (2024)
- [From Code to Prediction: Fine-Tuning LLMs for Neural Network Performance Classification in NNGPT](https://mlsanity.com/paper/2605.03686/) (2026)

## Abstract (arXiv)

With the booming demand for machine learning applications, it has been recognized that the number of knowledgeable data scientists can not scale with the growing data volumes and application needs in our digital world. In response to this demand, several automated machine learning (AutoML) frameworks have been developed to fill the gap of human expertise by automating the process of building machine learning pipelines. Each framework comes with different heuristics-based design decisions. In this study, we present a comprehensive evaluation and comparison of the performance characteristics of six popular AutoML frameworks, namely, AutoWeka, AutoSKlearn, TPOT, Recipe, ATM, and SmartML, across 100 data sets from established AutoML benchmark suites. Our experimental evaluation considers different aspects for its comparison, including the performance impact of several design decisions, including time budget, size of search space, meta-learning, and ensemble construction. The results of our study reveal various interesting insights that can significantly guide and impact the design of AutoML frameworks.