mlsanityData to 7 Oct 2026

Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data

arXiv 2402.17644 ↗First posted 27 Feb 2024cs.CL cs.AIcited by 61

On the map: LLM reasoning → Causal reasoning in LLMs

Abstract (arXiv)

Quantitative reasoning is a critical skill to analyze data, yet the assessment of such ability remains limited. To address this gap, we introduce the Quantitative Reasoning with Data (QRData) benchmark, aiming to evaluate Large Language Models' capability in statistical and causal reasoning with real-world data. The benchmark comprises a carefully constructed dataset of 411 questions accompanied by data sheets from textbooks, online learning materials, and academic papers. To compare models' quantitative reasoning abilities on data and text, we enrich the benchmark with an auxiliary set of 290 text-only questions, namely QRText. We evaluate natural language reasoning, program-based reasoning, and agent reasoning methods including Chain-of-Thought, Program-of-Thoughts, ReAct, and code interpreter assistants on diverse models. The strongest model GPT-4 achieves an accuracy of 58%, which has much room for improvement. Among open-source models, Deepseek-coder-instruct, a code LLM pretrained on 2T tokens, gets the highest accuracy of 37%. Analysis reveals that models encounter difficulties in data analysis and causal reasoning, and struggle in using causal knowledge and provided data simultaneously. Code and data are in https://github.com/xxxiaol/QRData.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning. Builds on 40 of its 60 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

40 of 60 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023The Magic of IF: Investigating Causal Reasoning Abilities in Large Language Models of Code30 May
  2. 2023Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation12 May
  3. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causality28 Apr

Other references · 37

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  10. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  11. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  12. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
25 more
  1. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  2. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  3. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  4. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  5. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  6. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  7. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  8. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  9. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  10. 2019Abductive Commonsense Reasoningcited by 300
  11. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
  12. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  13. 2021D'ya like DAGs? A Survey on Structure Learning and Causal Discoverycited by 209
  14. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 189
  15. 2023Can Large Language Models Infer Causation from Correlation?cited by 142
  16. 2023TableLlama: Towards Open Large Generalist Models for Tablescited by 136
  17. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 134
  18. 2022MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Datacited by 120
  19. 2021HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generationcited by 117
  20. 2019Counterfactual Story Reasoning and Generationcited by 87
  21. 2023Is GPT-4 a Good Data Analyst?cited by 69
  22. 2021Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Datacited by 43
  23. 2023FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domainscited by 39
  24. 2019Neuropathic Pain Diagnosis Simulator for Causal Discovery Algorithm Evaluationcited by 32
  25. 2022Execution-based Evaluation for Data Science Code Generation Modelscited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 28; 2026: 13.

61 papers.

Sort:

By area:

  1. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 111
  2. 2024DiscoveryBench: Towards Data-Driven Discovery with Large Language Modelscited by 97
  3. 2024BLADE: Benchmarking Language Model Agents for Data-Driven Sciencecited by 72
  4. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  5. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  6. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  7. 2024Are Large Language Models Good Statisticians?cited by 26
  8. 2024What Are the Odds? Language Models Are Capable of Probabilistic Reasoningcited by 25
  9. 2024Causality for Large Language Modelscited by 24
  10. 2024Aviary: training language agents on challenging scientific taskscited by 23
  11. 2025FoNE: Precise Single-Token Number Embeddings via Fourier Featurescited by 21
  12. 2024A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Educationcited by 16
38 more
  1. 2025MarketSenseAI 2.0: Enhancing Stock Analysis through LLM Agentscited by 16
  2. 2024CLEAR: Can Language Models Really Understand Causal Graphs?cited by 14
  3. 2026DSGym: A Holistic Framework for Evaluating and Training Data Science Agentscited by 14
  4. 2024CausalEval: Towards Better Causal Reasoning in Language Modelscited by 13
  5. 2025EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routingcited by 13
  6. 2025Learning from Diverse Reasoning Paths with Routing and Collaborationcited by 12
  7. 2025Scaling Generalist Data-Analytic Agentscited by 11
  8. 2024Causal Agent based on Large Language Modelcited by 10
  9. 2024C2P: Featuring Large Language Models with Causal Reasoningcited by 8
  10. 2025StatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysiscited by 8
  11. 2025A Survey on Enhancing Causal Reasoning Ability of Large Language Modelscited by 8
  12. 2025Reasoning Under Uncertainty: Exploring Probabilistic Reasoning Capabilities of LLMscited by 8
  13. 2025CoTSRF: Utilize Chain of Thought as Stealthy and Robust Fingerprint of Large Language Modelscited by 7
  14. 2025LLM Cannot Discover Causality, and Should Be Restricted to Non-Decisional Support in Causal Discoverycited by 7
  15. 2025Can LLMs Reconcile Knowledge Conflicts in Counterfactual Reasoningcited by 7
  16. 2026DataSTORM: Deep Research on Large-Scale Databases using Exploratory Data Analysis and Data Storytellingcited by 7
  17. 2024DACO: Towards Application-Driven and Comprehensive Data Analysis via Code Generationcited by 6
  18. 2024Evaluating Interventional Reasoning Capabilities of Large Language Modelscited by 6
  19. 2024CoverBench: A Challenging Benchmark for Complex Claim Verificationcited by 6
  20. 2024Data Analysis in the Era of Generative AIcited by 6
  21. 2025Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic Empirical Studycited by 6
  22. 2025Ice Cream Doesn't Cause Drowning: Benchmarking LLMs Against Statistical Pitfalls in Causal Inferencecited by 5
  23. 2025Code Execution as Grounded Supervision for LLM Reasoningcited by 5
  24. 2025AIRepr: An Analyst-Inspector Framework for Evaluating Reproducibility of LLMs in Data Sciencecited by 4
  25. 2025DSMentor: Enhancing Data Science Agents with Curriculum Learning and Online Knowledge Accumulationcited by 4
  26. 2025StatEval: A Comprehensive Benchmark for Large Language Models in Statisticscited by 4
  27. 2026CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimationcited by 4
  28. 2025Does visualization help AI understand data?cited by 3
  29. 2025Causal Reasoning in Pieces: Modular In-Context Learning for Causal Discoverycited by 3
  30. 2025When Dimensionality Hurts: The Role of LLM Embedding Compression for Noisy Regression Taskscited by 2
  31. 2025MULTITAT: Benchmarking Multilingual Table-and-Text Question Answeringcited by 2
  32. 2025A Financial Brain Scan of the LLMcited by 2
  33. 2026Scaling Scientific Discovery Environments for Turn-Level Agentic RLcited by 2
  34. 2024Can Large Language Models put 2 and 2 together? Probing for Entailed Arithmetical Relationshipscited by 1
  35. 2025Augmenting Research Ideation with Data: An Empirical Investigation in Social Sciencecited by 1
  36. 2025Towards Probabilistic Question Answering Over Tabular Datacited by 1
  37. 2025DSBC : Data Science task Benchmarking with Context engineeringcited by 1
  38. 2025Mini Amusement Parks (MAPs): A Testbed for Modelling Business Decisionscited by 1

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Can LLM Reasoning Be Trusted? A Comparative Study: Using Human Benchmarking on Statistical Tasks
  2. 2024CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models
  3. 2024A Critical Review of Causal Reasoning Benchmarks for Large Language Models
  4. 2026The CompMath-MCQ Dataset: Are LLMs Ready for Higher-Level Math?
  5. 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning
  6. 2026Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals
  7. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  8. 2026mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?