# StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMs

arXiv: https://arxiv.org/abs/2606.22977 · HTML: https://mlsanity.com/paper/2606.22977/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 22 Jun 2026.

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Can Large Language Models Infer Causation from Correlation?](https://mlsanity.com/paper/2306.05836/) (2023)
- [DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?](https://mlsanity.com/paper/2409.07703/) (2024)
- [CLadder: Assessing Causal Reasoning in Language Models](https://mlsanity.com/paper/2312.04350/) (2023)
- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data](https://mlsanity.com/paper/2402.17644/) (2024)
- [DataSciBench: An LLM Agent Benchmark for Data Science](https://mlsanity.com/paper/2502.13897/) (2025)
- [Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation](https://mlsanity.com/paper/2502.17521/) (2025)
- [Are Large Language Models Good Statisticians?](https://mlsanity.com/paper/2406.07815/) (2024)
- [An Overview of Large Language Models for Statisticians](https://mlsanity.com/paper/2502.17814/) (2025)
- [DCR: Quantifying Data Contamination in LLMs Evaluation](https://mlsanity.com/paper/2507.11405/) (2025)
- [StatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysis](https://mlsanity.com/paper/2502.17657/) (2025)
- [StatEval: A Comprehensive Benchmark for Large Language Models in Statistics](https://mlsanity.com/paper/2510.09517/) (2025)
- [DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems](https://mlsanity.com/paper/2601.13591/) (2026)
- [ModelingAgent: Bridging LLMs and Mathematical Modeling for Real-World Challenges](https://mlsanity.com/paper/2505.15068/) (2025)
- [EvalAssist: A Human-Centered Tool for LLM-as-a-Judge](https://mlsanity.com/paper/2507.02186/) (2025)

## Abstract (arXiv)

Statistical analysis is a broad, complex field requiring both domain knowledge and tool proficiency. While prior work has evaluated large language models (LLMs) in this domain, existing benchmarks remain limited in scope and format. To bridge this gap, we introduce StatABench (Statistical AnalysisBenchmark), a benchmark designed to systematically assess LLMs' statistical analysis capabilities. StatABench comprises two complementary components: Stat-Closed, containing 404 questions across 18 statistical topics in multiple formats (multiple-choice, fill-in-the-blank, decision-making, and practical application), and Stat-Open, featuring 30 complex open-ended modeling tasks adapted from professional competitions. We evaluate diverse LLMs using the LangChain MCP framework and multiple data science agents, and assess Stat-Open solutions via a validated LLM-as-Judge protocol. Experiments show that even GPT-5.1 achieves only 68.6% on Stat-Closed, while the best open-source model reaches 60.6%. On Stat-Open, the top agent framework scores 61.86 on average. These results reveal the gap between current LLMs and reliable statistical analysis, highlighting persistent challenges in tool-grounded reasoning, methodological decision-making, and end-to-end statistical modeling.