mlsanityData to 7 Oct 2026

CLadder: Assessing Causal Reasoning in Language Models

arXiv 2312.04350 ↗First posted 7 Dec 2023cs.CL cs.AI cs.LGcited by 102

On the map: LLM reasoning → Causal reasoning in LLMs

Abstract (arXiv)

The ability to perform causal reasoning is widely considered a core feature of intelligence. In this work, we investigate whether large language models (LLMs) can coherently reason about causality. Much of the existing work in natural language processing (NLP) focuses on evaluating commonsense causal reasoning in LLMs, thus failing to assess whether a model can perform causal inference in accordance with a set of well-defined formal rules. To address this, we propose a new NLP task, causal inference in natural language, inspired by the "causal inference engine" postulated by Judea Pearl et al. We compose a large dataset, CLadder, with 10K samples: based on a collection of causal graphs and queries (associational, interventional, and counterfactual), we obtain symbolic questions and ground-truth answers, through an oracle causal inference engine. These are then translated into natural language. We evaluate multiple LLMs on our dataset, and we introduce and evaluate a bespoke chain-of-thought prompting strategy, CausalCoT. We show that our task is highly challenging for LLMs, and we conduct an in-depth analysis to gain deeper insights into the causal reasoning abilities of LLMs. Our data is open-sourced at https://huggingface.co/datasets/causalNLP/cladder, and our code can be found at https://github.com/causalNLP/cladder.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning. Builds on 35 of its 100 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

35 of 100 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causality28 Apr

Other references · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
22 more
  1. 2019How Can We Know What Language Models Know?cited by 953
  2. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  3. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  4. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  5. 2019Abductive Commonsense Reasoningcited by 300
  6. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  7. 2023Causal Parrots: Large Language Models May Talk Causality But Are Not Causalcited by 146
  8. 2017Fair Inference On Outcomescited by 143
  9. 2023Can Large Language Models Infer Causation from Correlation?cited by 142
  10. 2016Foundations of Structural Causal Models with Cycles and Latent Variablescited by 134
  11. 2019Counterfactual Story Reasoning and Generationcited by 87
  12. 2018Event2Mind: Commonsense Inference on Events, Intents, and Reactionscited by 82
  13. 2015External Validity: From Do-Calculus to Transportability Across Populationscited by 75
  14. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  15. 2020Reasoning about Goals, Steps, and Temporal Ordering with WikiHowcited by 66
  16. 2023Understanding Causality with Large Language Models: Feasibility and Opportunitiescited by 53
  17. 2018Causal Reasoning for Algorithmic Fairnesscited by 51
  18. 2022Causal Fairness Analysiscited by 42
  19. 2021COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentencescited by 40
  20. 2021TellMeWhy: A Dataset for Answering Why-Questions in Narrativescited by 27
  21. 2022WikiWhy: Answering and Explaining Cause-and-Effect Questionscited by 21
  22. 2023Testing GPT-4 with Wolfram Alpha and Code Interpreter plug-ins on math and science problemscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 43; 2026: 35.

102 papers.

Sort:

By area:

  1. 2025Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Modelscited by 61
  2. 2025Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?cited by 36
  3. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustmentcited by 29
  4. 2024Large Language Models for Causal Discovery: Current Landscape and Future Directionscited by 25
  5. 2024Causality for Large Language Modelscited by 24
  6. 2024Cause and Effect: Can Large Language Models Truly Understand Causality?cited by 23
  7. 2024EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoningcited by 21
  8. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  9. 2024Counterfactual Token Generation in Large Language Modelscited by 14
  10. 2024CausalGraph2LLM: Evaluating LLMs for Causal Queriescited by 12
  11. 2024From Explainable to Interpretable Deep Learning for Natural Language Processing in Healthcare: How Far from Reality?cited by 11
  12. 2024Risks and Opportunities of Open-Source Generative AIcited by 11
38 more
  1. 2025Triangulating LLM Progress through Benchmarks, Games, and Cognitive Testscited by 11
  2. 2025BASIL: Bayesian Assessment of Sycophancy in LLMscited by 11
  3. 2025Epistemological Fault Lines Between Human and Artificial Intelligencecited by 11
  4. 2024CELLO: Causal Evaluation of Large Vision-Language Modelscited by 10
  5. 2024Causal Agent based on Large Language Modelcited by 10
  6. 2025CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstentioncited by 10
  7. 2025Executable Counterfactuals: Improving LLMs' Causal Reasoning Through Codecited by 10
  8. 2024CHECKWHY: Causal Fact Verification via Argument Structurecited by 9
  9. 2024Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answerscited by 9
  10. 2024C2P: Featuring Large Language Models with Causal Reasoningcited by 8
  11. 2025A Survey on Enhancing Causal Reasoning Ability of Large Language Modelscited by 8
  12. 2024Assessing the Zero-Shot Capabilities of LLMs for Action Evaluation in RLcited by 7
  13. 2024Mining Causality: AI-Assisted Search for Instrumental Variablescited by 7
  14. 2024Prompting Strategies for Enabling Large Language Models to Infer Causation from Correlationcited by 7
  15. 2025LLM Cannot Discover Causality, and Should Be Restricted to Non-Decisional Support in Causal Discoverycited by 7
  16. 2025Can LLMs Reconcile Knowledge Conflicts in Counterfactual Reasoningcited by 7
  17. 2024LLM-initialized Differentiable Causal Discoverycited by 6
  18. 2025Agentic LLM Framework for Adaptive Decision Discoursecited by 6
  19. 2025Unveiling and Causalizing CoT: A Causal Pespectivecited by 6
  20. 2025AdaSTaR: Adaptive Data Sampling for Training Self-Taught Reasonerscited by 6
  21. 2024Simulating Policy Impacts: Developing a Generative Scenario Writing Method to Evaluate the Perceived Effects of Regulationcited by 5
  22. 2024The Odyssey of Commonsense Causality: From Foundational Benchmarks to Cutting-Edge Reasoningcited by 5
  23. 2025Ice Cream Doesn't Cause Drowning: Benchmarking LLMs Against Statistical Pitfalls in Causal Inferencecited by 5
  24. 2025Mitigating Spurious Correlations in LLMs via Causality-Aware Post-Trainingcited by 5
  25. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  26. 2025Punctuation and Predicates in Language Modelscited by 5
  27. 2025Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Modelscited by 5
  28. 2026LLMs are not (consistently) Bayesian: Quantifying internal (in)consistencies of LLMs' probabilistic beliefscited by 5
  29. 2025Do Large Language Models Reason Causally Like Us? Even Better?cited by 4
  30. 2025Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Modelscited by 4
  31. 2025Can Large Language Models Infer Causal Relationships from Real-World Text?cited by 4
  32. 2025CausalARC: Abstract Reasoning with Causal World Modelscited by 4
  33. 2025PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detectioncited by 4
  34. 2026Recurrent Confidence Chain: Temporal-Aware Uncertainty Quantification in Large Language Modelscited by 4
  35. 2026CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimationcited by 4
  36. 2024Prompt2DeModel: Declarative Neuro-Symbolic Modeling with Natural Languagecited by 3
  37. 2025Algorithmic causal structure emerging through compressioncited by 3
  38. 2025ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Modelscited by 3

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Models and Causal Inference in Collaboration: A Survey
  2. 2025Causality for Natural Language Processing
  3. 2024A Critical Review of Causal Reasoning Benchmarks for Large Language Models
  4. 2024Causal Inference with Large Language Model: A Survey
  5. 2024COLD: Causal reasOning in cLosed Daily activities
  6. 2026NoisyCausal: A Benchmark for Evaluating Causal Reasoning Under Structured Noise
  7. 2024CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs
  8. 2025Causal Reasoning in Pieces: Modular In-Context Learning for Causal Discovery