mlsanityData to 14 Oct 2026

Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition

arXiv 2407.02651 ↗First posted 2 Jul 2024cs.HC cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

LLM-powered tools like ChatGPT Data Analysis, have the potential to help users tackle the challenging task of data analysis programming, which requires expertise in data processing, programming, and statistics. However, our formative study (n=15) uncovered serious challenges in verifying AI-generated results and steering the AI (i.e., guiding the AI system to produce the desired output). We developed two contrasting approaches to address these challenges. The first (Stepwise) decomposes the problem into step-by-step subgoals with pairs of editable assumptions and code until task completion, while the second (Phasewise) decomposes the entire problem into three editable, logical phases: structured input/output assumptions, execution plan, and code. A controlled, within-subjects experiment (n=18) compared these systems against a conversational baseline. Users reported significantly greater control with the Stepwise and Phasewise systems, and found intervention, correction, and verification easier, compared to the baseline. The results suggest design guidelines and trade-offs for AI-assisted data analysis tools.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 31 of its 103 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

31 of 103 references.

References · 31

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 253
  3. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 218
  4. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  5. 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
  6. 2022Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programmingcited by 91
  7. 2023The Metacognitive Demands and Opportunities of Generative AIcited by 87
  8. 2024CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needscited by 84
  9. 2023CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classescited by 72
  10. 2020How do Data Science Workers Collaborate? Roles, Workflows, and Toolscited by 64
  11. 2022"Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interactioncited by 63
  12. 2023"What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Modelscited by 48
19 more
  1. 2023Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Modelscited by 48
  2. 2020Conceptual Metaphors Impact Perceptions of Human-AI Collaborationcited by 47
  3. 2022Investigating Explainability of Generative AI for Code through Scenario-based Designcited by 47
  4. 2022What is it like to program with artificial intelligence?cited by 39
  5. 2021AutoDS: Towards Human-Centered Automation of Data Sciencecited by 32
  6. 2023On the Design of AI-powered Code Assistants for Notebookscited by 28
  7. 2023How Do Analysts Understand and Verify AI-Assisted Data Analyses?cited by 26
  8. 2023How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Studycited by 24
  9. 2022How do you Converse with an Analytical Chatbot? Revisiting Gricean Maxims for Designing Analytical Conversational Behaviorcited by 20
  10. 2023Interactive Text-to-SQL Generation via Editable Step-by-Step Explanationscited by 19
  11. 2022Aligning Offline Metrics and Human Judgments of Value for Code Generation Modelscited by 14
  12. 2022Is explainable AI a race against model complexity?cited by 9
  13. 2023Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunitiescited by 9
  14. 2019Paths Explored, Paths Omitted, Paths Obscured: Decision Points & Selective Reporting in End-to-End Data Analysiscited by 7
  15. 2019Talking datasets: Understanding data sensemaking behaviourscited by 7
  16. 2023CoLadder: Supporting Programmers with Hierarchical Code Generation in Multi-Level Abstractioncited by 7
  17. 2022Visualizing the Scripts of Data Wrangling with SOMNUScited by 4
  18. 2023Should Computers Be Easy To Use? Questioning the Doctrine of Simplicity in User Interface Designcited by 3
  19. 2022How Do Data Science Workers Communicate Intermediate Results?cited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 13; 2026: 11.

29 papers.

Sort:

By area:

  1. 2024WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualizationcited by 39
  2. 2024Cocoa: Co-Planning and Co-Execution with AI Agentscited by 39
  3. 2024Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learningcited by 16
  4. 2024LLM4DS: Evaluating Large Language Models for Data Science Code Generationcited by 7
  5. 2025Understanding, Protecting, and Augmenting Human Cognition with Generative AI: A Synthesis of the CHI 2025 Tools for Thought Workshopcited by 7
  6. 2024ChainBuddy: An AI Agent System for Generating LLM Pipelinescited by 6
  7. 2025When Should Users Check? Modeling Confirmation Frequency in Multi-Step Agentic AI Taskscited by 6
  8. 2025Human-AI Experience in Integrated Development Environments: A Systematic Literature Reviewcited by 5
  9. 2025Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Softwarecited by 5
  10. 2025Orca: Browsing at Scale Through User-Driven and AI-Facilitated Orchestration Across Malleable Webpagescited by 5
  11. 2025"The Diagram is like Guardrails": Structuring GenAI-assisted Hypotheses Exploration with an Interactive Shared Representationcited by 4
  12. 2025Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scalecited by 3
17 more
  1. 2025On the Regulatory Potential of User Interfaces for AI Agent Governancecited by 3
  2. 2026Overseeing Agents Without Constant Oversight: Challenges and Opportunitiescited by 3
  3. 2025Interactive Program Synthesis for Modeling Collaborative Physical Activities from Narrated Demonstrationscited by 2
  4. 2025HiLDe: Intentional Code Generation via Human-in-the-Loop Decodingcited by 1
  5. 2025LAFA: Agentic LLM-Driven Federated Analytics over Decentralized Data Sourcescited by 1
  6. 2026Dr. Claw: An AI Scientist Workspace for Vibe Researchcited by 1
  7. 2025VIDEE: Visual and Interactive Decomposition, Execution, and Evaluation of Text Analytics with Intelligent Agentscited by 0
  8. 2025OrchVis: Hierarchical Multi-Agent Orchestration for Human Oversightcited by 0
  9. 2026Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Makingcited by 0
  10. 2026From Binary Groundedness to Support Relations: Towards a Reader-Centred Taxonomy for Comprehension of AI Outputcited by 0
  11. 2026From Overload to Insights: How AI Agents Can Support Scientists in Analyzing Complex Datacited by 0
  12. 2026Plans Work in Mysterious Ways: Evaluating a Plan Mode for Spreadsheet Agentscited by 0
  13. 2026MUSE: An Interactive Meta-Agent for Understanding and Steering LLM-powered Data Science Systemscited by 0
  14. 2026AdaLens: Interactive Storyline for Monitoring and Steering Long-Running Agentic Data Analysiscited by 0
  15. 2026Exploring Agentic Approaches for Data Issue Detection and Repair in AI-Assisted Visualizationcited by 0
  16. 2026AI Soccer Analyst: Stage-Aware and Verifiable Human-AI Collaboration for Soccer Data Analysiscited by 0
  17. 2026ForkSCOPE: Charting the Agentic Garden of Forking Pathscited by 0