mlsanityData to 7 Oct 2026

Fits and Starts: Enterprise Use of AutoML and the Role of Humans in the Loop

arXiv 2101.04296 ↗First posted 12 Jan 2021cs.HC cs.CY cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

AutoML systems can speed up routine data science work and make machine learning available to those without expertise in statistics and computer science. These systems have gained traction in enterprise settings where pools of skilled data workers are limited. In this study, we conduct interviews with 29 individuals from organizations of different sizes to characterize how they currently use, or intend to use, AutoML systems in their data science work. Our investigation also captures how data visualization is used in conjunction with AutoML systems. Our findings identify three usage scenarios for AutoML that resulted in a framework summarizing the level of automation desired by data workers with different levels of expertise. We surfaced the tension between speed and human oversight and found that data visualization can do a poor job balancing the two. Our findings have implications for the design and implementation of human-in-the-loop visual analytics approaches.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 15 of its 50 references that are in the map. First posted on arXiv 12 Jan 2021.

Builds on

15 of 50 references.

References · 15

  1. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  2. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 251
  3. 2019Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AIcited by 79
  4. 2016Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Sciencecited by 73
  5. 2020Human Factors in Model Interpretability: Industry Practices, Challenges, and Needscited by 73
  6. 2020How do Data Science Workers Collaborate? Roles, Workflows, and Toolscited by 64
  7. 2018Automated Machine Learning: From Principles to Practicescited by 62
  8. 2020A Survey of Visual Analytics Techniques for Machine Learningcited by 56
  9. 2020Trust in AutoML: Exploring Information Needs for Establishing Trust in Automated Machine Learning Systemscited by 36
  10. 2019AutoAIViz: Opening the Blackbox of Automated Artificial Intelligence with Conditional Parallel Coordinatescited by 24
  11. 2019ATMSeer: Increasing Transparency and Controllability in Automated Machine Learningcited by 23
  12. 2020PipelineProfiler: A Visual Analytics Tool for the Exploration of AutoML Pipelinescited by 19
3 more
  1. 2020Soliciting Human-in-the-Loop User Feedback for Interactive Machine Learning Reduces User Trust and Impressions of Model Accuracycited by 19
  2. 2020Surfacing Visualization Miragescited by 17
  3. 2020Data Science: A Comprehensive Overviewcited by 6

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 4; 2023: 7; 2024: 4; 2025: 3.

21 papers.

Sort:

  1. 2022Interactive Model Cards: A Human-Centered Approach to Model Documentationcited by 61
  2. 2023Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoMLcited by 16
  3. 2024Automated data processing and feature engineering for deep learning and big data applications: a surveycited by 14
  4. 2024Position: A Call to Action for a Human-Centered AutoML Paradigmcited by 14
  5. 2023Where Are We So Far? Understanding Data Storytelling Tools from the Perspective of Human-AI Collaborationcited by 13
  6. 2021Naive Automated Machine Learningcited by 8
  7. 2022XAutoML: A Visual Analytics Tool for Understanding and Validating Automated Machine Learningcited by 8
  8. 2023AutoML in The Wild: Obstacles, Workarounds, and Expectationscited by 8
  9. 2023Why is AI not a Panacea for Data Workers? An Interview Study on Human-AI Collaboration in Data Storytellingcited by 7
  10. 2022An Empirical Study on the Usage of Automated Machine Learning Toolscited by 4
  11. 2023Assessing the Use of AutoML for Data-Driven Software Engineeringcited by 2
  12. 2021Naive Automated Machine Learning -- A Late Baseline for AutoMLcited by 1
9 more
  1. 2022Practitioner Motives to Use Different Hyperparameter Optimization Methodscited by 1
  2. 2023DREAM: Debugging and Repairing AutoML Pipelinescited by 1
  3. 2025Data-Prompt Co-Evolution: Growing Test Sets to Refine LLM Behaviorcited by 1
  4. 2021Man versus Machine: AutoML and Human Experts' Role in Phishing Detectioncited by 0
  5. 2023Eliciting Model Steering Interactions from Users via Data and Visual Design Probescited by 0
  6. 2024Human-Centered AI Product Prototyping with No-Code AutoML: Conceptual Framework, Potentials and Limitationscited by 0
  7. 2024Behavior Matters: An Alternative Perspective on Promoting Responsible Data Sciencecited by 0
  8. 2025Medical artificial intelligence toolbox (MAIT): an explainable machine learning framework for binary classification, survival modelling, and regression analysescited by 0
  9. 2025Survey for Categorising Explainable AI Studies Using Data Analysis Task Frameworkscited by 0