mlsanityData to 7 Oct 2026

StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMs

arXiv 2606.22977 ↗First posted 22 Jun 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Statistical analysis is a broad, complex field requiring both domain knowledge and tool proficiency. While prior work has evaluated large language models (LLMs) in this domain, existing benchmarks remain limited in scope and format. To bridge this gap, we introduce StatABench (Statistical AnalysisBenchmark), a benchmark designed to systematically assess LLMs' statistical analysis capabilities. StatABench comprises two complementary components: Stat-Closed, containing 404 questions across 18 statistical topics in multiple formats (multiple-choice, fill-in-the-blank, decision-making, and practical application), and Stat-Open, featuring 30 complex open-ended modeling tasks adapted from professional competitions. We evaluate diverse LLMs using the LangChain MCP framework and multiple data science agents, and assess Stat-Open solutions via a validated LLM-as-Judge protocol. Experiments show that even GPT-5.1 achieves only 68.6% on Stat-Closed, while the best open-source model reaches 60.6%. On Stat-Open, the top agent framework scores 61.86 on average. These results reveal the gap between current LLMs and reliable statistical analysis, highlighting persistent challenges in tool-grounded reasoning, methodological decision-making, and end-to-end statistical modeling.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 42 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

26 of 42 references.

References · 26

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2023Can Large Language Models Infer Causation from Correlation?cited by 142
14 more
  1. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?cited by 119
  2. 2023CLadder: Assessing Causal Reasoning in Language Modelscited by 102
  3. 2024Large Language Models Meet NLP: A Surveycited by 102
  4. 2024Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Datacited by 61
  5. 2025DataSciBench: An LLM Agent Benchmark for Data Sciencecited by 53
  6. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  7. 2024Are Large Language Models Good Statisticians?cited by 26
  8. 2025An Overview of Large Language Models for Statisticianscited by 24
  9. 2025DCR: Quantifying Data Contamination in LLMs Evaluationcited by 10
  10. 2025StatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysiscited by 8
  11. 2025EvalAssist: A Human-Centered Tool for LLM-as-a-Judgecited by 5
  12. 2026DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problemscited by 5
  13. 2025ModelingAgent: Bridging LLMs and Mathematical Modeling for Real-World Challengescited by 4
  14. 2025StatEval: A Comprehensive Benchmark for Large Language Models in Statisticscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Can LLM Reasoning Be Trusted? A Comparative Study: Using Human Benchmarking on Statistical Tasks
  2. 2026Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities
  3. 2024FinDABench: Benchmarking Financial Data Analysis Ability of Large Language Models
  4. 2025CLEAR: Error Analysis via LLM-as-a-Judge Made Easy
  5. 2026ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
  6. 2025IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis
  7. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
  8. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data