# Are Large Language Models Good Statisticians?

arXiv: https://arxiv.org/abs/2406.07815 · HTML: https://mlsanity.com/paper/2406.07815/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 70 references that are in the map.
- First posted on arXiv 12 Jun 2024.

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning](https://mlsanity.com/paper/2209.14610/) (2022)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [TheoremQA: A Theorem-driven Question Answering dataset](https://mlsanity.com/paper/2305.12524/) (2023)
- [What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks](https://mlsanity.com/paper/2305.18365/) (2023)
- [InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks](https://mlsanity.com/paper/2401.05507/) (2024)
- [A Survey of Deep Learning for Mathematical Reasoning](https://mlsanity.com/paper/2212.10535/) (2022)
- [Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance](https://mlsanity.com/paper/2305.17306/) (2023)
- [NaturalProofs: Mathematical Theorem Proving in Natural Language](https://mlsanity.com/paper/2104.01112/) (2021)
- [The Dawn of Natural Language to SQL: Are We Fully Ready?](https://mlsanity.com/paper/2406.01265/) (2024)
- [Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data](https://mlsanity.com/paper/2402.17644/) (2024)
- [Towards Natural Language Interfaces for Data Visualization: A Survey](https://mlsanity.com/paper/2109.03506/) (2021)

## Built on by (most cited first, 27 of 27)

- [Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning](https://mlsanity.com/paper/2505.14069/) (2025)
- [Assemble Your Crew: Automatic Multi-agent Communication Topology Design via Autoregressive Graph Generation](https://mlsanity.com/paper/2507.18224/) (2025)
- [A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?](https://mlsanity.com/paper/2408.05109/) (2024)
- [Self-Supervised Prompt Optimization](https://mlsanity.com/paper/2502.06855/) (2025)
- [A Survey of Data Agents: Emerging Paradigm or Overstated Hype?](https://mlsanity.com/paper/2510.23587/) (2025)
- [HAIChart: Human and AI Paired Visualization System](https://mlsanity.com/paper/2406.11033/) (2024)
- [Reasoning RAG via System 1 or System 2: A Survey on Reasoning Agentic Retrieval-Augmented Generation for Industry Challenges](https://mlsanity.com/paper/2506.10408/) (2025)
- [MEBench: Benchmarking Large Language Models for Cross-Document Multi-Entity Question Answering](https://mlsanity.com/paper/2502.18993/) (2025)
- [EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routing](https://mlsanity.com/paper/2503.22402/) (2025)
- [StatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysis](https://mlsanity.com/paper/2502.17657/) (2025)
- [DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoning](https://mlsanity.com/paper/2508.01700/) (2025)
- [FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes](https://mlsanity.com/paper/2506.03278/) (2025)
- [StatEval: A Comprehensive Benchmark for Large Language Models in Statistics](https://mlsanity.com/paper/2510.09517/) (2025)
- [DataPuzzle: Breaking Free from the Hallucinated Promise of LLMs in Data Analysis](https://mlsanity.com/paper/2504.10036/) (2025)
- [Automated Model Discovery via Multi-modal & Multi-step Pipeline](https://mlsanity.com/paper/2509.25946/) (2025)
- [Scaling Scientific Discovery Environments for Turn-Level Agentic RL](https://mlsanity.com/paper/2607.28990/) (2026)
- [Fishing for Answers: Exploring One-shot vs. Iterative Retrieval Strategies for Retrieval Augmented Generation](https://mlsanity.com/paper/2509.04820/) (2025)
- [MDSF: Context-Aware Multi-Dimensional Data Storytelling Framework based on Large language Model](https://mlsanity.com/paper/2501.01014/) (2025)
- [A-ProS: Towards Reliable Autonomous Programming Through Multi-Model Feedback](https://mlsanity.com/paper/2605.18073/) (2026)
- [TuneAhead: Predicting Fine-tuning Performance Before Full Training Begins](https://mlsanity.com/paper/2606.17660/) (2026)
- [SQLConductor: Search-to-Policy Learning for Step-wise Text-to-SQL Orchestration](https://mlsanity.com/paper/2606.23537/) (2026)
- [A Risk Decomposition Framework for Pre-Hoc Fine-Tuning Prediction](https://mlsanity.com/paper/2606.17649/) (2026)
- [StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMs](https://mlsanity.com/paper/2606.22977/) (2026)
- [Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing](https://mlsanity.com/paper/2608.07437/) (2026)
- [AgentFactory: Towards Automated Agentic System Design and Optimization](https://mlsanity.com/paper/2609.01045/) (2026)
- [Benchmarking Language Models for Statistical Problem Formulation](https://mlsanity.com/paper/2609.01982/) (2026)
- [ACED-Bench: Evaluating How LLM Agents Acquire and Act on Causal Evidence](https://mlsanity.com/paper/2610.14381/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated impressive capabilities across a range of scientific tasks including mathematics, physics, and chemistry. Despite their successes, the effectiveness of LLMs in handling complex statistical tasks remains systematically under-explored. To bridge this gap, we introduce StatQA, a new benchmark designed for statistical analysis tasks. StatQA comprises 11,623 examples tailored to evaluate LLMs' proficiency in specialized statistical tasks and their applicability assessment capabilities, particularly for hypothesis testing methods. We systematically experiment with representative LLMs using various prompting strategies and show that even state-of-the-art models such as GPT-4o achieve a best performance of only 64.83%, indicating significant room for improvement. Notably, while open-source LLMs (e.g. LLaMA-3) show limited capability, those fine-tuned ones exhibit marked improvements, outperforming all in-context learning-based methods (e.g. GPT-4o). Moreover, our comparative human experiments highlight a striking contrast in error types between LLMs and humans: LLMs primarily make applicability errors, whereas humans mostly make statistical task confusion errors. This divergence highlights distinct areas of proficiency and deficiency, suggesting that combining LLM and human expertise could lead to complementary strengths, inviting further investigation into their collaborative potential. Our source code and data are available at https://statqa.github.io/.