CausalT5k: Diagnosing Refusal and Failure Modes in Trustworthy Causal Reasoning Across Causal Rungs
On the map: LLM reasoning → Causal reasoning in LLMs
Abstract (arXiv)
Large language models increasingly produce fluent causal explanations, yet they often fail in ways aggregate accuracy cannot diagnose: confusing association with intervention, abandoning correct judgments under pressure, over-refusing valid claims, or answering when evidence is underdetermined. We introduce CTK, a diagnostic benchmark of 5,147 cases and growing, across 10 domains and all three levels of Pearl's Ladder of Causation. Unlike benchmarks that only score correctness, CTK reveals why a model failed by annotating causal rung, trap type, pressure sensitivity, refusal quality, and Utility-Safety tradeoffs. Its Sheep/Wolf taxonomy separates valid causal designs from inferential traps; paired neutral/pressure variants measure sycophantic drift through Bad Flip Rate; and Wise Refusal fields test whether a model identifies the missing information needed before endorsing a claim. CTK exposes failure modes hidden by aggregate accuracy: the Skepticism Trap, Rung Collapse under scaling, pressure-induced drift, Detection-Correction gaps, and counterfactual error modes. Rather than prescribing a correction method, it provides the diagnostic substrate for studying causal-reasoning failure profiles.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning. Builds on 17 of its 36 references that are in the map. First posted on arXiv 9 Feb 2026.
Builds on
17 of 36 references.
Closest ancestors · 4
Same direction, newest first
- 2026Diagnosing and Mitigating Sycophancy and Skepticism in LLM Causal Judgment13 Jan
- 2025Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?26 Jun
- 2023CLadder: Assessing Causal Reasoning in Language Models7 Dec
- 2022e-CARE: a New Dataset for Exploring Explainable Causal Reasoning12 May
Other references · 13
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
- 2023Can Large Language Models Infer Causation from Correlation?cited by 142
- 2021CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Modelscited by 33
- 2025SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulascited by 30
- 2024Linear Probe Penalties Reduce LLM Sycophancycited by 26
- 2024LLMScan: Causal Scan for LLM Misbehavior Detectioncited by 13
- 2024Causal Reasoning in Large Language Models: A Knowledge Graph Approachcited by 9
- 2024Language Agents Meet Causality -- Bridging LLMs and Causal World Modelscited by 8
- 2024Scalable and Flexible Causal Discovery with an Efficient Test for Adjacencycited by 5
- 2025A Checks-and-Balances Framework for Context-Aware Ethical AI Alignmentcited by 3
1 more
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Uncovering Hidden Correctness in LLM Causal Reasoning via Symbolic Verification
- 2024A Critical Review of Causal Reasoning Benchmarks for Large Language Models
- 2025Ice Cream Doesn't Cause Drowning: Benchmarking LLMs Against Statistical Pitfalls in Causal Inference
- 2026The Illusion of textitWhat If: Evaluating the Breakdown of Counterfactual Reasoning in LLMs
- 2026NoisyCausal: A Benchmark for Evaluating Causal Reasoning Under Structured Noise
- 2024CHECKWHY: Causal Fact Verification via Argument Structure
- 2026Better Think Thrice: Learning to Reason Causally with Double Counterfactual Consistency
- 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causality