mlsanityData to 7 Oct 2026

WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization

arXiv 2408.01703 ↗First posted 3 Aug 2024cs.HCcited by 38

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language models (LLMs) support data analysis through conversational user interfaces, as exemplified in OpenAI's ChatGPT (formally known as Advanced Data Analysis or Code Interpreter). Essentially, LLMs produce code for accomplishing diverse analysis tasks. However, presenting raw code can obscure the logic and hinder user verification. To empower users with enhanced comprehension and augmented control over analysis conducted by LLMs, we propose a novel approach to transform LLM-generated code into an interactive visual representation. In the approach, users are provided with a clear, step-by-step visualization of the LLM-generated code in real time, allowing them to understand, verify, and modify individual data operations in the analysis. Our design decisions are informed by a formative study (N=8) probing into user practice and challenges. We further developed a prototype named WaitGPT and conducted a user study (N=12) to evaluate its usability and effectiveness. The findings from the user study reveal that WaitGPT facilitates monitoring and steering of data analysis performed by LLMs, enabling participants to enhance error detection and increase their overall confidence in the results.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 37 of its 78 references that are in the map. First posted on arXiv 3 Aug 2024.

Builds on

37 of 78 references.

References · 37

  1. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  2. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  3. 2023Using an LLM to Help With Code Understandingcited by 222
  4. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 191
  5. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  6. 2023The Metacognitive Demands and Opportunities of Generative AIcited by 85
  7. 2023Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfacescited by 71
  8. 2023ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Languagecited by 71
  9. 2023Is GPT-4 a Good Data Analyst?cited by 69
  10. 2023Graphologue: Exploring Large Language Model Responses with Interactive Diagramscited by 65
  11. 2021Towards Natural Language Interfaces for Data Visualization: A Surveycited by 58
  12. 2023"What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Modelscited by 48
25 more
  1. 2023Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Modelscited by 48
  2. 2024Designing a Dashboard for Transparency and Control of Conversational AIcited by 43
  3. 2023Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queriescited by 38
  4. 2024DynaVis: Dynamically Synthesized UI Widgets for Visualization Editingcited by 34
  5. 2023DirectGPT: A Direct Manipulation Interface to Interact with Large Language Modelscited by 32
  6. 2023Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Promptscited by 30
  7. 2024Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decompositioncited by 29
  8. 2023On the Design of AI-powered Code Assistants for Notebookscited by 28
  9. 2024Automated Data Visualization from Natural Language via Large Language Models: An Exploratory Studycited by 26
  10. 2023How Do Analysts Understand and Verify AI-Assisted Data Analyses?cited by 25
  11. 2023How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Studycited by 24
  12. 2023AVA: Towards Autonomous Visualization Agents through Visual Perception-Driven Decision-Makingcited by 21
  13. 2023XNLI: Explaining and Diagnosing NLI-based Visual Data Analysiscited by 18
  14. 2023Beyond Generating Code: Evaluating GPT on a Data Visualization Coursecited by 16
  15. 2023Low-code LLM: Graphical User Interface over Large Language Modelscited by 13
  16. 2024An AI-Resilient Text Rendering Technique for Reading and Skimming Documentscited by 10
  17. 2023Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunitiescited by 8
  18. 2023Creating Emordle: Animating Word Cloud for Emotion Expressioncited by 7
  19. 2023Wakey-Wakey: Animate Text by Mimicking Characters in a GIFcited by 7
  20. 2023WonderFlow: Narration-Centric Design of Animated Data Videoscited by 7
  21. 2024Sketch Then Generate: Providing Incremental User Feedback and Guiding LLM Code Generation through Language-Oriented Code Sketchescited by 7
  22. 2023CrossData: Leveraging Text-Data Connections for Authoring Data Documentscited by 6
  23. 2022Smallset Timelines: A Visual Representation of Data Preprocessing Decisionscited by 4
  24. 2022Visualizing the Scripts of Data Wrangling with SOMNUScited by 4
  25. 2023Urania: Visualizing Data Analysis Pipelines for Natural Language-Based Data Explorationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 19; 2026: 18.

38 papers.

Sort:

By area:

  1. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 167
  2. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
  3. 2024A Survey on Large Language Model-based Agents for Statistics and Data Sciencecited by 22
  4. 2025Large Language Model-based Data Science Agent: A Surveycited by 22
  5. 2025VisCoder: Fine-Tuning LLMs for Executable Python Visualization Code Generationcited by 13
  6. 2025Agentic Visualization: Extracting Agent-based Design Patterns from Visualization Systemscited by 12
  7. 2025DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoningcited by 8
  8. 2026Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agentscited by 6
  9. 2025RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generationcited by 4
  10. 2025"The Diagram is like Guardrails": Structuring GenAI-assisted Hypotheses Exploration with an Interactive Shared Representationcited by 3
  11. 2025NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modificationcited by 3
  12. 2025Automated Model Discovery via Multi-modal & Multi-step Pipelinecited by 3
26 more
  1. 2025From Clicks to Conversations: Evaluating the Effectiveness of Conversational Agents in Statistical Analysiscited by 2
  2. 2025Interactive Program Synthesis for Modeling Collaborative Physical Activities from Narrated Demonstrationscited by 2
  3. 2025CodeWatcher: IDE Telemetry Data Extraction Tool for Understanding Coding Interactions with LLMscited by 2
  4. 2025NoteEx: Interactive Visual Context Manipulation for LLM-Assisted Exploratory Data Analysis in Computational Notebookscited by 2
  5. 2026ProMAS: Proactive Error Forecasting for Multi-Agent Systems Using Markov Transition Dynamicscited by 2
  6. 2026"What Did It Actually Do?": Understanding Risk Awareness and Traceability for Computer-Use Agentscited by 2
  7. 2025HiLDe: Intentional Code Generation via Human-in-the-Loop Decodingcited by 1
  8. 2025Challenges & Opportunities with LLM-Assisted Visualization Retargetingcited by 1
  9. 2026"When to Hand Off, When to Work Together": Understanding Concurrent Human-Agent Interaction in Shared Co-Creative Workspacescited by 1
  10. 2026AwesomeLit: Towards Hypothesis Generation with Agent-Supported Literature Researchcited by 1
  11. 2026InconLens: Interactive Visual Diagnosis of Behavioral Inconsistencies in LLM-based Agentic Systemscited by 1
  12. 2026Auditing and Controlling AI Agent Actions in Spreadsheetscited by 1
  13. 2026ATOM: Instantiating Budget-Controllable Multi-Agent Collaboration via Nucleus-Electron Hierarchycited by 1
  14. 2026TwinBI: An Agentic Digital Twin for Efficient Augmented Interactions with Business Intelligence Dashboardscited by 1
  15. 2026Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Modelcited by 1
  16. 2025VIDEE: Visual and Interactive Decomposition, Execution, and Evaluation of Text Analytics with Intelligent Agentscited by 0
  17. 2025AutoBridge: Automating Smart Device Integration with Centralized Platformcited by 0
  18. 2025In-Place Feedback: Reliable Refinement for Multi-Turn Expert-LLM Collaborationcited by 0
  19. 2026Within the MDT Room: Situated in Multidisciplinary Team-Grounded Agent Debate for Clinical Diagnosiscited by 0
  20. 2026Mixed-Initiative Context: Structuring and Managing Context for Human-AI Collaborationcited by 0
  21. 2026MV-Bench: Benchmarking Multimodal Large Language Models for Coordinated Multi-View Interface Constructioncited by 0
  22. 2026AdaLens: Interactive Storyline for Monitoring and Steering Long-Running Agentic Data Analysiscited by 0
  23. 2026Exploring Agentic Approaches for Data Issue Detection and Repair in AI-Assisted Visualizationcited by 0
  24. 2026Dr. Claw: An AI Scientist Workspace for Vibe Researchcited by 0
  25. 2026AI Soccer Analyst: Stage-Aware and Verifiable Human-AI Collaboration for Soccer Data Analysiscited by 0
  26. 2026Point, Revise, Review: Grounded Agentic Analysis in Reactive Notebooks with marimo-lenscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024InsightLens: Augmenting LLM-Powered Data Analysis with Interactive Insight Management and Navigation
  2. 2025IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis
  3. 2024Testing and Evaluation of Large Language Models: Correctness, Non-Toxicity, and Fairness
  4. 2025Illuminating LLM Coding Agents: Visual Analytics for Deeper Understanding and Enhancement
  5. 2024Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agents
  6. 2024ChatGPT Alternative Solutions: Large Language Models Survey
  7. 2023ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?
  8. 2025ViseGPT: Towards Better Alignment of LLM-generated Data Wrangling Scripts and User Prompts