# CausalT5k: Diagnosing Refusal and Failure Modes in Trustworthy Causal Reasoning Across Causal Rungs

arXiv: https://arxiv.org/abs/2602.08939 · HTML: https://mlsanity.com/paper/2602.08939/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning.
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Placed in

- LLM reasoning / [Causal reasoning in LLMs](https://mlsanity.com/lr/causal-reasoning-in-llms/)

## Builds on (most cited first, 17 of 17)

- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Inverse Scaling: When Bigger Isn't Better](https://mlsanity.com/paper/2306.09479/) (2023)
- [Can Large Language Models Infer Causation from Correlation?](https://mlsanity.com/paper/2306.05836/) (2023)
- [CLadder: Assessing Causal Reasoning in Language Models](https://mlsanity.com/paper/2312.04350/) (2023)
- [e-CARE: a New Dataset for Exploring Explainable Causal Reasoning](https://mlsanity.com/paper/2205.05849/) (2022)
- [Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?](https://mlsanity.com/paper/2506.21215/) (2025)
- [CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models](https://mlsanity.com/paper/2112.11941/) (2021)
- [SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulas](https://mlsanity.com/paper/2505.14615/) (2025)
- [Linear Probe Penalties Reduce LLM Sycophancy](https://mlsanity.com/paper/2412.00967/) (2024)
- [LLMScan: Causal Scan for LLM Misbehavior Detection](https://mlsanity.com/paper/2410.16638/) (2024)
- [Causal Reasoning in Large Language Models: A Knowledge Graph Approach](https://mlsanity.com/paper/2410.11588/) (2024)
- [Language Agents Meet Causality -- Bridging LLMs and Causal World Models](https://mlsanity.com/paper/2410.19923/) (2024)
- [Scalable and Flexible Causal Discovery with an Efficient Test for Adjacency](https://mlsanity.com/paper/2406.09177/) (2024)
- [Diagnosing and Mitigating Sycophancy and Skepticism in LLM Causal Judgment](https://mlsanity.com/paper/2601.08258/) (2026)
- [A Checks-and-Balances Framework for Context-Aware Ethical AI Alignment](https://mlsanity.com/paper/2502.00136/) (2025)
- [AGI Requires a Coordination Layer on Top of Pattern Repositories](https://mlsanity.com/paper/2512.05765/) (2025)

## Built on by (most cited first, 2 of 2)

- [TRACE: An Operational Reasoning Schema for Auditable Agentic Commitments](https://mlsanity.com/paper/2607.12480/) (2026)
- [CoBe: A Benchmark For Conversational Counterfactual Text Editing](https://mlsanity.com/paper/2610.13057/) (2026)

## Abstract (arXiv)

Large language models increasingly produce fluent causal explanations, yet they often fail in ways aggregate accuracy cannot diagnose: confusing association with intervention, abandoning correct judgments under pressure, over-refusing valid claims, or answering when evidence is underdetermined. We introduce CTK, a diagnostic benchmark of 5,147 cases and growing, across 10 domains and all three levels of Pearl's Ladder of Causation. Unlike benchmarks that only score correctness, CTK reveals why a model failed by annotating causal rung, trap type, pressure sensitivity, refusal quality, and Utility-Safety tradeoffs. Its Sheep/Wolf taxonomy separates valid causal designs from inferential traps; paired neutral/pressure variants measure sycophantic drift through Bad Flip Rate; and Wise Refusal fields test whether a model identifies the missing information needed before endorsing a claim. CTK exposes failure modes hidden by aggregate accuracy: the Skepticism Trap, Rung Collapse under scaling, pressure-induced drift, Detection-Correction gaps, and counterfactual error modes. Rather than prescribing a correction method, it provides the diagnostic substrate for studying causal-reasoning failure profiles.