# Is GPT-4 a Good Data Analyst?

arXiv: https://arxiv.org/abs/2305.15038 · HTML: https://mlsanity.com/paper/2305.15038/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 22 of 22)

- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation](https://mlsanity.com/paper/1905.08205/) (2019)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases](https://mlsanity.com/paper/1909.05378/) (2019)
- [Chart-to-Text: A Large-Scale Benchmark for Chart Summarization](https://mlsanity.com/paper/2203.06486/) (2022)
- [SParC: Cross-Domain Semantic Parsing in Context](https://mlsanity.com/paper/1906.02285/) (2019)
- [Text Generation from Knowledge Graphs with Graph Transformers](https://mlsanity.com/paper/1904.02342/) (2019)
- [Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge](https://mlsanity.com/paper/1901.07931/) (2019)
- [RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQL](https://mlsanity.com/paper/2205.06983/) (2022)
- [Structure-informed Language Models Are Protein Designers](https://mlsanity.com/paper/2302.01649/) (2023)
- [Large Language Models are Diverse Role-Players for Summarization Evaluation](https://mlsanity.com/paper/2303.15078/) (2023)
- [AI-Generated Content (AIGC): A Survey](https://mlsanity.com/paper/2304.06632/) (2023)
- [A Short Survey of Viewing Large Language Models in Legal Aspect](https://mlsanity.com/paper/2303.09136/) (2023)
- [Evaluating Psychological Safety of Large Language Models](https://mlsanity.com/paper/2212.10529/) (2022)
- [Towards Generalizable and Robust Text-to-SQL Parsing](https://mlsanity.com/paper/2210.12674/) (2022)
- [Professional Certification Benchmark Dataset: The First 500 Jobs For Large Language Models](https://mlsanity.com/paper/2305.05377/) (2023)

## Built on by (most cited first, 30 of 70)

- [MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization](https://mlsanity.com/paper/2402.11453/) (2024)
- [Large Language Models and Games: A Survey and Roadmap](https://mlsanity.com/paper/2402.18659/) (2024)
- [Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow](https://mlsanity.com/paper/2306.07209/) (2023)
- [Unleashing the potential of prompt engineering for large language models](https://mlsanity.com/paper/2310.14735/) (2023)
- [Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data](https://mlsanity.com/paper/2402.17644/) (2024)
- [VisEval: A Benchmark for Data Visualization in the Era of Large Language Models](https://mlsanity.com/paper/2407.00981/) (2024)
- [Exploring the Potential of Large Language Models in Computational Argumentation](https://mlsanity.com/paper/2311.09022/) (2023)
- [WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization](https://mlsanity.com/paper/2408.01703/) (2024)
- [IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations](https://mlsanity.com/paper/2404.01266/) (2024)
- [LogGPT: Exploring ChatGPT for Log-Based Anomaly Detection](https://mlsanity.com/paper/2309.01189/) (2023)
- [Large Language Models as Data Preprocessors](https://mlsanity.com/paper/2308.16361/) (2023)
- [InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation](https://mlsanity.com/paper/2407.06423/) (2024)
- [DialogBench: Evaluating LLMs as Human-like Dialogue Systems](https://mlsanity.com/paper/2311.01677/) (2023)
- [Benchmarking Data Science Agents](https://mlsanity.com/paper/2402.17168/) (2024)
- [WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models](https://mlsanity.com/paper/2411.05451/) (2024)
- [HAIChart: Human and AI Paired Visualization System](https://mlsanity.com/paper/2406.11033/) (2024)
- [LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs](https://mlsanity.com/paper/2408.08656/) (2024)
- [TabLib: A Dataset of 627M Tables with Context](https://mlsanity.com/paper/2310.07875/) (2023)
- [Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions](https://mlsanity.com/paper/2501.04437/) (2025)
- [Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks](https://mlsanity.com/paper/2410.01428/) (2024)
- [A Survey on Large Language Model-based Agents for Statistics and Data Science](https://mlsanity.com/paper/2412.14222/) (2024)
- [Jellyfish: A Large Language Model for Data Preprocessing](https://mlsanity.com/paper/2312.01678/) (2023)
- [PlotGen: Multi-Agent LLM-based Scientific Data Visualization via Multimodal Feedback](https://mlsanity.com/paper/2502.00988/) (2025)
- [ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?](https://mlsanity.com/paper/2311.16989/) (2023)
- [The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective](https://mlsanity.com/paper/2407.08583/) (2024)
- [What Should Data Science Education Do with Large Language Models?](https://mlsanity.com/paper/2307.02792/) (2023)
- [AstroVisBench: A Code Benchmark for Scientific Computing and Visualization in Astronomy](https://mlsanity.com/paper/2505.20538/) (2025)
- [From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews](https://mlsanity.com/paper/2312.01202/) (2023)
- [Comparing large language models and human programmers for generating programming code](https://mlsanity.com/paper/2403.00894/) (2024)
- [Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factors](https://mlsanity.com/paper/2404.03304/) (2024)

## Abstract (arXiv)

As large language models (LLMs) have demonstrated their powerful capabilities in plenty of domains and tasks, including context understanding, code generation, language generation, data storytelling, etc., many data analysts may raise concerns if their jobs will be replaced by artificial intelligence (AI). This controversial topic has drawn great attention in public. However, we are still at a stage of divergent opinions without any definitive conclusion. Motivated by this, we raise the research question of "is GPT-4 a good data analyst?" in this work and aim to answer it by conducting head-to-head comparative studies. In detail, we regard GPT-4 as a data analyst to perform end-to-end data analysis with databases from a wide range of domains. We propose a framework to tackle the problems by carefully designing the prompts for GPT-4 to conduct experiments. We also design several task-specific evaluation metrics to systematically compare the performance between several professional human data analysts and GPT-4. Experimental results show that GPT-4 can achieve comparable performance to humans. We also provide in-depth discussions about our results to shed light on further studies before reaching the conclusion that GPT-4 can replace data analysts.