mlsanityData to 7 Oct 2026

Is GPT-4 a Good Data Analyst?

arXiv 2305.15038 ↗First posted 24 May 2023cs.CLcited by 69

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) have demonstrated their powerful capabilities in plenty of domains and tasks, including context understanding, code generation, language generation, data storytelling, etc., many data analysts may raise concerns if their jobs will be replaced by artificial intelligence (AI). This controversial topic has drawn great attention in public. However, we are still at a stage of divergent opinions without any definitive conclusion. Motivated by this, we raise the research question of "is GPT-4 a good data analyst?" in this work and aim to answer it by conducting head-to-head comparative studies. In detail, we regard GPT-4 as a data analyst to perform end-to-end data analysis with databases from a wide range of domains. We propose a framework to tackle the problems by carefully designing the prompts for GPT-4 to conduct experiments. We also design several task-specific evaluation metrics to systematically compare the performance between several professional human data analysts and GPT-4. Experimental results show that GPT-4 can achieve comparable performance to humans. We also provide in-depth discussions about our results to shed light on further studies before reaching the conclusion that GPT-4 can replace data analysts.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 22 of its 33 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

22 of 33 references.

References · 22

  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  2. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  3. 2023BloombergGPT: A Large Language Model for Financecited by 923
  4. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  5. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  6. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  7. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  8. 2019Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representationcited by 210
  9. 2022Is GPT-3 a Good Data Annotator?cited by 208
  10. 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 180
  11. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  12. 2019SParC: Cross-Domain Semantic Parsing in Contextcited by 141
10 more
  1. 2019Text Generation from Knowledge Graphs with Graph Transformerscited by 122
  2. 2019Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challengecited by 86
  3. 2022RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQLcited by 85
  4. 2023Structure-informed Language Models Are Protein Designerscited by 68
  5. 2023Large Language Models are Diverse Role-Players for Summarization Evaluationcited by 62
  6. 2023AI-Generated Content (AIGC): A Surveycited by 59
  7. 2023A Short Survey of Viewing Large Language Models in Legal Aspectcited by 51
  8. 2022Evaluating Psychological Safety of Large Language Modelscited by 26
  9. 2022Towards Generalizable and Robust Text-to-SQL Parsingcited by 7
  10. 2023Professional Certification Benchmark Dataset: The First 500 Jobs For Large Language Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 29; 2025: 17; 2026: 5.

69 papers.

Sort:

By area:

  1. 2024MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualizationcited by 98
  2. 2024Large Language Models and Games: A Survey and Roadmapcited by 91
  3. 2023Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflowcited by 76
  4. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  5. 2024Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Datacited by 61
  6. 2024VisEval: A Benchmark for Data Visualization in the Era of Large Language Modelscited by 43
  7. 2023Exploring the Potential of Large Language Models in Computational Argumentationcited by 42
  8. 2024WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualizationcited by 38
  9. 2024IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representationscited by 37
  10. 2023Large Language Models as Data Preprocessorscited by 34
  11. 2023LogGPT: Exploring ChatGPT for Log-Based Anomaly Detectioncited by 33
  12. 2023DialogBench: Evaluating LLMs as Human-like Dialogue Systemscited by 31
38 more
  1. 2024Benchmarking Data Science Agentscited by 30
  2. 2024InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generationcited by 30
  3. 2024WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Modelscited by 28
  4. 2024HAIChart: Human and AI Paired Visualization Systemcited by 26
  5. 2024LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMscited by 24
  6. 2023TabLib: A Dataset of 627M Tables with Contextcited by 23
  7. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  8. 2024Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Taskscited by 22
  9. 2024A Survey on Large Language Model-based Agents for Statistics and Data Sciencecited by 22
  10. 2023Jellyfish: A Large Language Model for Data Preprocessingcited by 20
  11. 2025PlotGen: Multi-Agent LLM-based Scientific Data Visualization via Multimodal Feedbackcited by 19
  12. 2023ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?cited by 15
  13. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
  14. 2023What Should Data Science Education Do with Large Language Models?cited by 13
  15. 2025AstroVisBench: A Code Benchmark for Scientific Computing and Visualization in Astronomycited by 13
  16. 2024Comparing large language models and human programmers for generating programming codecited by 9
  17. 2024Concept -- An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factorscited by 9
  18. 2023Conversational AI Threads for Visualizing Multidimensional Datasetscited by 8
  19. 2023From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviewscited by 8
  20. 2024FashionReGen: LLM-Empowered Fashion Report Generationcited by 8
  21. 2024EconNLI: Evaluating Large Language Models on Economics Reasoningcited by 8
  22. 2025DatawiseAgent: A Notebook-Centric LLM Agent Framework for Adaptive and Robust Data Science Automationcited by 8
  23. 2025SPIO: Ensemble and Selective Strategies via LLM-Based Multi-Agent Planning in Automated Data Sciencecited by 7
  24. 2024DACO: Towards Application-Driven and Comprehensive Data Analysis via Code Generationcited by 6
  25. 2024Data Analysis in the Era of Generative AIcited by 6
  26. 2023Co-audit: tools to help humans double-check AI-generated contentcited by 5
  27. 2023Language Models for Business Optimisation with a Real World Case Study in Production Schedulingcited by 4
  28. 2023"I'd Like to Have an Argument, Please": Argumentative Reasoning in Large Language Modelscited by 4
  29. 2023Capture the Flag: Uncovering Data Insights with Large Language Modelscited by 4
  30. 2024An Empirical Study of Challenges in Machine Learning Asset Managementcited by 4
  31. 2024XForecast: Evaluating Natural Language Explanations for Time Series Forecastingcited by 4
  32. 2025DSMentor: Enhancing Data Science Agents with Curriculum Learning and Online Knowledge Accumulationcited by 4
  33. 2025Measuring Data Science Automation: A Survey of Evaluation Tools for AI Assistants and Agentscited by 4
  34. 2026MultiVis-Agent: A Multi-Agent Framework with Logic Rules for Reliable and Comprehensive Cross-Modal Data Visualizationcited by 3
  35. 2023AutoPCF: Efficient Product Carbon Footprint Accounting with Large Language Modelscited by 2
  36. 2024Charting the Future: Using Chart Question-Answering for Scalable Evaluation of LLM-Driven Data Visualizationscited by 2
  37. 2025ReSpark: Leveraging Previous Data Reports as References to Generate New Reports with LLMscited by 2
  38. 2024ParaICL: Towards Parallel In-Context Learningcited by 1

50 most cited of 69 All 69 (JSON)