# How Do Analysts Understand and Verify AI-Assisted Data Analyses?

arXiv: https://arxiv.org/abs/2309.10947 · HTML: https://mlsanity.com/paper/2309.10947/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2023.

## Built on by (most cited first, 26 of 26)

- [VisEval: A Benchmark for Data Visualization in the Era of Large Language Models](https://mlsanity.com/paper/2407.00981/) (2024)
- [WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization](https://mlsanity.com/paper/2408.01703/) (2024)
- [Beyond the Chat: Executable and Verifiable Text-Editing with LLMs](https://mlsanity.com/paper/2309.15337/) (2023)
- [The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmers](https://mlsanity.com/paper/2404.02806/) (2024)
- [Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition](https://mlsanity.com/paper/2407.02651/) (2024)
- [Data Formulator 2: Iterative Creation of Data Visualizations, with AI Transforming Data Along the Way](https://mlsanity.com/paper/2408.16119/) (2024)
- [Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learning](https://mlsanity.com/paper/2410.08922/) (2024)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [Data Analysis in the Era of Generative AI](https://mlsanity.com/paper/2409.18475/) (2024)
- [Capture the Flag: Uncovering Data Insights with Large Language Models](https://mlsanity.com/paper/2312.13876/) (2023)
- [AIRepr: An Analyst-Inspector Framework for Evaluating Reproducibility of LLMs in Data Science](https://mlsanity.com/paper/2502.16395/) (2025)
- [Humans are Missing from AI Coding Agent Research](https://mlsanity.com/paper/2608.12355/) (2026)
- [Charting the Future: Using Chart Question-Answering for Scalable Evaluation of LLM-Driven Data Visualizations](https://mlsanity.com/paper/2409.18764/) (2024)
- [More Than "Means to an End": Supporting Reasoning with Transparently Designed AI Data Science Processes](https://mlsanity.com/paper/2603.24877/) (2026)
- ["What Did It Actually Do?": Understanding Risk Awareness and Traceability for Computer-Use Agents](https://mlsanity.com/paper/2603.28551/) (2026)
- [From PDFs to Structured Data: Utilizing LLM Analysis in Sports Database Management](https://mlsanity.com/paper/2410.17619/) (2024)
- [A Review on Large Language Models for Visual Analytics](https://mlsanity.com/paper/2503.15176/) (2025)
- [Auditing and Controlling AI Agent Actions in Spreadsheets](https://mlsanity.com/paper/2604.20070/) (2026)
- [Dr. Claw: An AI Scientist Workspace for Vibe Research](https://mlsanity.com/paper/2609.00365/) (2026)
- [Mixed-Initiative Context: Structuring and Managing Context for Human-AI Collaboration](https://mlsanity.com/paper/2604.07121/) (2026)
- [Human agency in initial human-AI proof formalization workflows](https://mlsanity.com/paper/2606.04273/) (2026)
- [AdaLens: Interactive Storyline for Monitoring and Steering Long-Running Agentic Data Analysis](https://mlsanity.com/paper/2608.17834/) (2026)
- [Exploring Agentic Approaches for Data Issue Detection and Repair in AI-Assisted Visualization](https://mlsanity.com/paper/2608.21602/) (2026)
- [AI Soccer Analyst: Stage-Aware and Verifiable Human-AI Collaboration for Soccer Data Analysis](https://mlsanity.com/paper/2609.11224/) (2026)
- [ForkSCOPE: Charting the Agentic Garden of Forking Paths](https://mlsanity.com/paper/2609.12438/) (2026)
- [DataWeave: Deploying Human-LLM Analytics for Exploratory Structured Data Analysis](https://mlsanity.com/paper/2610.02679/) (2026)

## Abstract (arXiv)

Data analysis is challenging as it requires synthesizing domain knowledge, statistical expertise, and programming skills. Assistants powered by large language models (LLMs), such as ChatGPT, can assist analysts by translating natural language instructions into code. However, AI-assistant responses and analysis code can be misaligned with the analyst's intent or be seemingly correct but lead to incorrect conclusions. Therefore, validating AI assistance is crucial and challenging. Here, we explore how analysts understand and verify the correctness of AI-generated analyses. To observe analysts in diverse verification approaches, we develop a design probe equipped with natural language explanations, code, visualizations, and interactive data tables with common data operations. Through a qualitative user study (n=22) using this probe, we uncover common behaviors within verification workflows and how analysts' programming, analysis, and tool backgrounds reflect these behaviors. Additionally, we provide recommendations for analysts and highlight opportunities for designers to improve future AI-assistant experiences.