# How Much Automation Does a Data Scientist Want?

arXiv: https://arxiv.org/abs/2101.03970 · HTML: https://mlsanity.com/paper/2101.03970/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 139 references that are in the map.
- First posted on arXiv 7 Jan 2021.

## Builds on (most cited first, 30 of 35)

- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [Metrics for Explainable AI: Challenges and Prospects](https://mlsanity.com/paper/1812.04608/) (2018)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy](https://mlsanity.com/paper/2002.04087/) (2020)
- [One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques](https://mlsanity.com/paper/1909.03012/) (2019)
- [Learning to Complement Humans](https://mlsanity.com/paper/2005.00582/) (2020)
- [Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Making](https://mlsanity.com/paper/1902.02960/) (2019)
- [HoloClean: Holistic Data Repairs with Probabilistic Inference](https://mlsanity.com/paper/1702.00820/) (2017)
- [Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI](https://mlsanity.com/paper/1909.02309/) (2019)
- [Feature Engineering for Predictive Modeling using Reinforcement Learning](https://mlsanity.com/paper/1709.07150/) (2017)
- [How do Data Science Workers Collaborate? Roles, Workflows, and Tools](https://mlsanity.com/paper/2001.06684/) (2020)
- [Conscientious Classification: A Data Scientist's Guide to Discrimination-Aware Classification](https://mlsanity.com/paper/1907.09013/) (2019)
- [An ADMM Based Framework for AutoML Pipeline Configuration](https://mlsanity.com/paper/1905.00424/) (2019)
- [One button machine for automating feature engineering in relational databases](https://mlsanity.com/paper/1706.00327/) (2017)
- [Probabilistic Matrix Factorization for Automated Machine Learning](https://mlsanity.com/paper/1705.05355/) (2017)
- [How Data Scientists Work Together With Domain Experts in Scientific Collaborations: To Find The Right Answer Or To Ask The Right Question?](https://mlsanity.com/paper/1909.03486/) (2019)
- [AutoAIViz: Opening the Blackbox of Automated Artificial Intelligence with Conditional Parallel Coordinates](https://mlsanity.com/paper/1912.06723/) (2019)
- [ATMSeer: Increasing Transparency and Controllability in Automated Machine Learning](https://mlsanity.com/paper/1902.05009/) (2019)
- [AlphaClean: Automatic Generation of Data Cleaning Pipelines](https://mlsanity.com/paper/1904.11827/) (2019)
- [Benchmark and Survey of Automated Machine Learning Frameworks](https://mlsanity.com/paper/1904.12054/) (2019)
- [Selecting Near-Optimal Learners via Incremental Data Allocation](https://mlsanity.com/paper/1601.00024/) (2015)
- [Easy Hyperparameter Search Using Optunity](https://mlsanity.com/paper/1412.1114/) (2014)
- [Discovering the Sweet Spot of Human-Computer Configurations: A Case Study in Information Extraction](https://mlsanity.com/paper/1909.07065/) (2019)

## Built on by (most cited first, 20 of 20)

- [Natural Language to Code Generation in Interactive Data Science Notebooks](https://mlsanity.com/paper/2212.09248/) (2022)
- [Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation](https://mlsanity.com/paper/2204.11788/) (2022)
- [Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits](https://mlsanity.com/paper/2205.06922/) (2022)
- [AutoDS: Towards Human-Centered Automation of Data Science](https://mlsanity.com/paper/2101.05273/) (2021)
- [How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study](https://mlsanity.com/paper/2101.06098/) (2021)
- [Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Function](https://mlsanity.com/paper/2406.01382/) (2024)
- [Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences](https://mlsanity.com/paper/2310.04621/) (2023)
- [Telling Stories from Computational Notebooks: AI-Assisted Presentation Slides Creation for Presenting Data Science Work](https://mlsanity.com/paper/2203.11085/) (2022)
- [Position: A Call to Action for a Human-Centered AutoML Paradigm](https://mlsanity.com/paper/2406.03348/) (2024)
- [Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoML](https://mlsanity.com/paper/2104.04375/) (2021)
- [Large Language Models Synergize with Automated Machine Learning](https://mlsanity.com/paper/2405.03727/) (2024)
- [XAutoML: A Visual Analytics Tool for Understanding and Validating Automated Machine Learning](https://mlsanity.com/paper/2202.11954/) (2022)
- [AutoML in The Wild: Obstacles, Workarounds, and Expectations](https://mlsanity.com/paper/2302.10827/) (2023)
- [Why is AI not a Panacea for Data Workers? An Interview Study on Human-AI Collaboration in Data Storytelling](https://mlsanity.com/paper/2304.08366/) (2023)
- [HAConvGNN: Hierarchical Attention Based Convolutional Graph Neural Network for Code Documentation Generation in Jupyter Notebooks](https://mlsanity.com/paper/2104.01002/) (2021)
- [An Empirical Study on the Usage of Automated Machine Learning Tools](https://mlsanity.com/paper/2208.13116/) (2022)
- [Human-Centered AI for Data Science: A Systematic Approach](https://mlsanity.com/paper/2110.01108/) (2021)
- [Practitioner Motives to Use Different Hyperparameter Optimization Methods](https://mlsanity.com/paper/2203.01717/) (2022)
- [A Survey on Semantics in Automated Data Science](https://mlsanity.com/paper/2205.08018/) (2022)
- [AutoQML: A Framework for Automated Quantum Machine Learning](https://mlsanity.com/paper/2502.21025/) (2025)

## Abstract (arXiv)

Data science and machine learning (DS/ML) are at the heart of the recent advancements of many Artificial Intelligence (AI) applications. There is an active research thread in AI, autoai, that aims to develop systems for automating end-to-end the DS/ML Lifecycle. However, do DS and ML workers really want to automate their DS/ML workflow? To answer this question, we first synthesize a human-centered AutoML framework with 6 User Role/Personas, 10 Stages and 43 Sub-Tasks, 5 Levels of Automation, and 5 Types of Explanation, through reviewing research literature and marketing reports. Secondly, we use the framework to guide the design of an online survey study with 217 DS/ML workers who had varying degrees of experience, and different user roles "matching" to our 6 roles/personas. We found that different user personas participated in distinct stages of the lifecycle -- but not all stages. Their desired levels of automation and types of explanation for AutoML also varied significantly depending on the DS/ML stage and the user persona. Based on the survey results, we argue there is no rationale from user needs for complete automation of the end-to-end DS/ML lifecycle. We propose new next steps for user-controlled DS/ML automation.