mlsanityData to 14 Oct 2026

How Much Automation Does a Data Scientist Want?

arXiv 2101.03970 ↗First posted 7 Jan 2021cs.LG cs.HCcited by 20

Not on the map. Why?

Abstract (arXiv)

Data science and machine learning (DS/ML) are at the heart of the recent advancements of many Artificial Intelligence (AI) applications. There is an active research thread in AI, autoai, that aims to develop systems for automating end-to-end the DS/ML Lifecycle. However, do DS and ML workers really want to automate their DS/ML workflow? To answer this question, we first synthesize a human-centered AutoML framework with 6 User Role/Personas, 10 Stages and 43 Sub-Tasks, 5 Levels of Automation, and 5 Types of Explanation, through reviewing research literature and marketing reports. Secondly, we use the framework to guide the design of an online survey study with 217 DS/ML workers who had varying degrees of experience, and different user roles "matching" to our 6 roles/personas. We found that different user personas participated in distinct stages of the lifecycle -- but not all stages. Their desired levels of automation and types of explanation for AutoML also varied significantly depending on the DS/ML stage and the user persona. Based on the survey results, we argue there is no rationale from user needs for complete automation of the end-to-end DS/ML lifecycle. We propose new next steps for user-controlled DS/ML automation.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 35 of its 139 references that are in the map. First posted on arXiv 7 Jan 2021.

Builds on

35 of 139 references.

References · 35

  1. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  2. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  3. 2018Model Cards for Model Reportingcited by 1,080
  4. 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
  5. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  6. 2016The Mythos of Model Interpretabilitycited by 721
  7. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  8. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 378
  9. 2018Metrics for Explainable AI: Challenges and Prospectscited by 260
  10. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 253
  11. 2020Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthycited by 183
  12. 2019One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniquescited by 164
23 more
  1. 2020Learning to Complement Humanscited by 124
  2. 2019Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Makingcited by 109
  3. 2017HoloClean: Holistic Data Repairs with Probabilistic Inferencecited by 79
  4. 2019Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AIcited by 79
  5. 2017Feature Engineering for Predictive Modeling using Reinforcement Learningcited by 78
  6. 2020How do Data Science Workers Collaborate? Roles, Workflows, and Toolscited by 64
  7. 2019Conscientious Classification: A Data Scientist's Guide to Discrimination-Aware Classificationcited by 55
  8. 2019An ADMM Based Framework for AutoML Pipeline Configurationcited by 47
  9. 2017One button machine for automating feature engineering in relational databasescited by 32
  10. 2017Probabilistic Matrix Factorization for Automated Machine Learningcited by 31
  11. 2019How Data Scientists Work Together With Domain Experts in Scientific Collaborations: To Find The Right Answer Or To Ask The Right Question?cited by 30
  12. 2019AutoAIViz: Opening the Blackbox of Automated Artificial Intelligence with Conditional Parallel Coordinatescited by 24
  13. 2019ATMSeer: Increasing Transparency and Controllability in Automated Machine Learningcited by 23
  14. 2019AlphaClean: Automatic Generation of Data Cleaning Pipelinescited by 23
  15. 2019Benchmark and Survey of Automated Machine Learning Frameworkscited by 18
  16. 2015Selecting Near-Optimal Learners via Incremental Data Allocationcited by 12
  17. 2014Easy Hyperparameter Search Using Optunitycited by 11
  18. 2019Discovering the Sweet Spot of Human-Computer Configurations: A Case Study in Information Extractioncited by 11
  19. 2020Data Vision: Learning to See Through Algorithmic Abstractioncited by 11
  20. 2019How can AI Automate End-to-End Data Science?cited by 7
  21. 2017The Many Faces of Data-centric Workflow Optimization: A Surveycited by 3
  22. 2018PANDA: Facilitating Usable AI Developmentcited by 2
  23. 2020Towards Automating the AI Operations Lifecyclecited by 2

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 8; 2023: 3; 2024: 3; 2025: 1.

20 papers.

Sort:

  1. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 87
  2. 2022Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderationcited by 67
  3. 2022Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkitscited by 34
  4. 2021AutoDS: Towards Human-Centered Automation of Data Sciencecited by 32
  5. 2021How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Studycited by 29
  6. 2024Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Functioncited by 24
  7. 2023Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiencescited by 21
  8. 2022Telling Stories from Computational Notebooks: AI-Assisted Presentation Slides Creation for Presenting Data Science Workcited by 15
  9. 2024Position: A Call to Action for a Human-Centered AutoML Paradigmcited by 14
  10. 2021Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoMLcited by 12
  11. 2024Large Language Models Synergize with Automated Machine Learningcited by 9
  12. 2022XAutoML: A Visual Analytics Tool for Understanding and Validating Automated Machine Learningcited by 8
8 more
  1. 2023AutoML in The Wild: Obstacles, Workarounds, and Expectationscited by 8
  2. 2023Why is AI not a Panacea for Data Workers? An Interview Study on Human-AI Collaboration in Data Storytellingcited by 7
  3. 2021HAConvGNN: Hierarchical Attention Based Convolutional Graph Neural Network for Code Documentation Generation in Jupyter Notebookscited by 6
  4. 2022An Empirical Study on the Usage of Automated Machine Learning Toolscited by 4
  5. 2021Human-Centered AI for Data Science: A Systematic Approachcited by 2
  6. 2022Practitioner Motives to Use Different Hyperparameter Optimization Methodscited by 1
  7. 2022A Survey on Semantics in Automated Data Sciencecited by 0
  8. 2025AutoQML: A Framework for Automated Quantum Machine Learningcited by 0