mlsanityData to 14 Oct 2026

AutoMLBench: A Comprehensive Experimental Evaluation of Automated Machine Learning Frameworks

arXiv 2204.08358 ↗First posted 18 Apr 2022cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

With the booming demand for machine learning applications, it has been recognized that the number of knowledgeable data scientists can not scale with the growing data volumes and application needs in our digital world. In response to this demand, several automated machine learning (AutoML) frameworks have been developed to fill the gap of human expertise by automating the process of building machine learning pipelines. Each framework comes with different heuristics-based design decisions. In this study, we present a comprehensive evaluation and comparison of the performance characteristics of six popular AutoML frameworks, namely, AutoWeka, AutoSKlearn, TPOT, Recipe, ATM, and SmartML, across 100 data sets from established AutoML benchmark suites. Our experimental evaluation considers different aspects for its comparison, including the performance impact of several design decisions, including time budget, size of search space, meta-learning, and ensemble construction. The results of our study reveal various interesting insights that can significantly guide and impact the design of AutoML frameworks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 14 of its 52 references that are in the map. First posted on arXiv 18 Apr 2022.

Builds on

14 of 52 references.

References · 14

  1. 2014OpenML: networked science in machine learningcited by 526
  2. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
  3. 2019AutoML: A Survey of the State-of-the-Artcited by 390
  4. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 383
  5. 2018Meta-Learning: A Surveycited by 256
  6. 2018Auto-Keras: An Efficient Neural Architecture Search Systemcited by 171
  7. 2019FLAML: A Fast and Lightweight AutoML Librarycited by 106
  8. 2019An Open Source AutoML Benchmarkcited by 85
  9. 2016Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Sciencecited by 75
  10. 2021LightAutoML: AutoML Solution for a Large Financial Services Ecosystemcited by 32
  11. 2019Automated Machine Learning: State-of-The-Art and Open Challengescited by 29
  12. 2019Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Toolscited by 25
2 more
  1. 2020GAMA: a General Automated Machine learning Assistantcited by 15
  2. 2019The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Developmentcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 1.

3 papers.

  1. 2024What fifty-one years of Linguistics and Artificial Intelligence research tell us about their correlation: A scientometric analysiscited by 8
  2. 2024GenoTEX: An LLM Agent Benchmark for Automated Gene Expression Data Analysiscited by 5
  3. 2026From Code to Prediction: Fine-Tuning LLMs for Neural Network Performance Classification in NNGPTcited by 0