# An Empirical Study on the Usage of Automated Machine Learning Tools

arXiv: https://arxiv.org/abs/2208.13116 · HTML: https://mlsanity.com/paper/2208.13116/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 47 references that are in the map.
- First posted on arXiv 28 Aug 2022.

## Builds on (most cited first, 16 of 16)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning](https://mlsanity.com/paper/2007.04074/) (2020)
- [Auto-Keras: An Efficient Neural Architecture Search System](https://mlsanity.com/paper/1806.10282/) (2018)
- [Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI](https://mlsanity.com/paper/1909.02309/) (2019)
- [Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science](https://mlsanity.com/paper/1603.06212/) (2016)
- [Automated Machine Learning: From Principles to Practices](https://mlsanity.com/paper/1810.13306/) (2018)
- [Trust in AutoML: Exploring Information Needs for Establishing Trust in Automated Machine Learning Systems](https://mlsanity.com/paper/2001.06509/) (2020)
- [Automated Machine Learning: State-of-The-Art and Open Challenges](https://mlsanity.com/paper/1906.02287/) (2019)
- [Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Tools](https://mlsanity.com/paper/1908.05557/) (2019)
- [VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition](https://mlsanity.com/paper/2107.08861/) (2021)
- [Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projects](https://mlsanity.com/paper/2002.03389/) (2020)
- [ATMSeer: Increasing Transparency and Controllability in Automated Machine Learning](https://mlsanity.com/paper/1902.05009/) (2019)
- [Whither AutoML? Understanding the Role of Automation in Machine Learning Workflows](https://mlsanity.com/paper/2101.04834/) (2021)
- [Fits and Starts: Enterprise Use of AutoML and the Role of Humans in the Loop](https://mlsanity.com/paper/2101.04296/) (2021)
- [How Much Automation Does a Data Scientist Want?](https://mlsanity.com/paper/2101.03970/) (2021)
- [Studying the Practices of Deploying Machine Learning Projects on Docker](https://mlsanity.com/paper/2206.00699/) (2022)

## Built on by (most cited first, 4 of 4)

- [Assessing the Use of AutoML for Data-Driven Software Engineering](https://mlsanity.com/paper/2307.10774/) (2023)
- [Improvement of Computational Performance of Evolutionary AutoML in a Heterogeneous Environment](https://mlsanity.com/paper/2301.05102/) (2023)
- [EDCA - An Evolutionary Data-Centric AutoML Framework for Efficient Pipelines](https://mlsanity.com/paper/2503.04350/) (2025)
- [What Causes Exceptions in Machine Learning Applications? Mining Machine Learning-Related Stack Traces on Stack Overflow](https://mlsanity.com/paper/2304.12857/) (2023)

## Abstract (arXiv)

The popularity of automated machine learning (AutoML) tools in different domains has increased over the past few years. Machine learning (ML) practitioners use AutoML tools to automate and optimize the process of feature engineering, model training, and hyperparameter optimization and so on. Recent work performed qualitative studies on practitioners' experiences of using AutoML tools and compared different AutoML tools based on their performance and provided features, but none of the existing work studied the practices of using AutoML tools in real-world projects at a large scale. Therefore, we conducted an empirical study to understand how ML practitioners use AutoML tools in their projects. To this end, we examined the top 10 most used AutoML tools and their respective usages in a large number of open-source project repositories hosted on GitHub. The results of our study show 1) which AutoML tools are mostly used by ML practitioners and 2) the characteristics of the repositories that use these AutoML tools. Also, we identified the purpose of using AutoML tools (e.g. model parameter sampling, search space management, model evaluation/error-analysis, Data/ feature transformation, and data labeling) and the stages of the ML pipeline (e.g. feature engineering) where AutoML tools are used. Finally, we report how often AutoML tools are used together in the same source code files. We hope our results can help ML practitioners learn about different AutoML tools and their usages, so that they can pick the right tool for their purposes. Besides, AutoML tool developers can benefit from our findings to gain insight into the usages of their tools and improve their tools to better fit the users' usages and needs.