mlsanityData to 7 Oct 2026

A Critical Review of Causal Reasoning Benchmarks for Large Language Models

arXiv 2407.08029 ↗First posted 10 Jul 2024cs.LG cs.CLcited by 13

On the map: LLM reasoning → Causal reasoning in LLMs

Abstract (arXiv)

Numerous benchmarks aim to evaluate the capabilities of Large Language Models (LLMs) for causal inference and reasoning. However, many of them can likely be solved through the retrieval of domain knowledge, questioning whether they achieve their purpose. In this review, we present a comprehensive overview of LLM benchmarks for causality. We highlight how recent benchmarks move towards a more thorough definition of causal reasoning by incorporating interventional or counterfactual reasoning. We derive a set of criteria that a useful benchmark or set of benchmarks should aim to satisfy. We hope this work will pave the way towards a general framework for the assessment of causal understanding in LLMs and the design of novel benchmarks.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning. Builds on 20 of its 28 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

20 of 28 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023The Magic of IF: Investigating Causal Reasoning Abilities in Large Language Models of Code30 May
  2. 2023Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation12 May
  3. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causality28 Apr
  4. 2022Towards Fine-grained Causal Reasoning and QA15 Apr

Other references · 16

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  3. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  4. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  5. 2023Challenges and Applications of Large Language Modelscited by 251
  6. 2023Causal Parrots: Large Language Models May Talk Causality But Are Not Causalcited by 146
  7. 2023Can Large Language Models Infer Causation from Correlation?cited by 142
  8. 2020Reasoning about Goals, Steps, and Temporal Ordering with WikiHowcited by 66
  9. 2023Understanding Causality with Large Language Models: Feasibility and Opportunitiescited by 53
  10. 2020WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challengecited by 43
  11. 2021COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentencescited by 40
  12. 2023Causal-Discovery Performance of ChatGPT in the context of Neuropathic Pain Diagnosiscited by 36
4 more
  1. 2021Guided Generation of Cause and Effectcited by 33
  2. 2021CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Modelscited by 33
  3. 2020Financial Document Causality Detection Shared Task (FinCausal 2020)cited by 31
  4. 2021TellMeWhy: A Dataset for Answering Why-Questions in Narrativescited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 8.

13 papers.

Sort:

By area:

  1. 2024Multi-Agent Causal Discovery Using Large Language Modelscited by 26
  2. 2025Agentic AI Needs a Systems Theorycited by 19
  3. 2026Lying with Truths: Open-Channel Multi-Agent Collusion for Belief Manipulation via Generative Montagecited by 17
  4. 2024CausalEval: Towards Better Causal Reasoning in Language Modelscited by 13
  5. 2025Position: Foundation Models Need Digital Twin Representationscited by 10
  6. 2024Do LLMs Have the Generalization Ability in Conducting Causal Inference?cited by 2
  7. 2026Epistemic Regret Minimization: Label-Free Causal Critique Beyond Outcome Rewardcited by 2
  8. 2026METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Modelscited by 1
  9. 2026CausalSmith: A Formally Grounded, Self-Improving Agentic Framework for Automated Research in Causal Inferencecited by 0
  10. 2026The Illusion of textitWhat If: Evaluating the Breakdown of Counterfactual Reasoning in LLMscited by 0
  11. 2026CausalVerify: End-to-End Verification of Causal Analyses by Language Modelscited by 0
  12. 2026Evidence-Grounded Auditing of Identification Assumptions in Climate-Policy Causal Evaluationscited by 0
1 more
  1. 2026Externalized CPDAG Summaries Improve LLM Causal Deductioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating Interventional Reasoning Capabilities of Large Language Models
  2. 2024CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs
  3. 2025A Survey on Enhancing Causal Reasoning Ability of Large Language Models
  4. 2025Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
  5. 2026CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching
  6. 2025Ice Cream Doesn't Cause Drowning: Benchmarking LLMs Against Statistical Pitfalls in Causal Inference
  7. 2024Large Language Models and Causal Inference in Collaboration: A Survey
  8. 2023CLadder: Assessing Causal Reasoning in Language Models