mlsanityData to 14 Oct 2026

The Illusion of textitWhat If: Evaluating the Breakdown of Counterfactual Reasoning in LLMs

arXiv 2608.27953 ↗First posted 28 Aug 2026cs.AIcited by 0

On the map: LLM reasoning → Causal reasoning in LLMs

Abstract (arXiv)

Counterfactual reasoning requires models to reason beyond the observed world and explain how altered conditions propagate through downstream consequences. Existing benchmarks largely target bounded settings with fixed variables or single gold outcomes, overlooking open-domain scenarios requiring causal-process evaluation. To this end, we present WhatIfBench, a diagnostic benchmark for open-domain, open-form, long-horizon counterfactual causal reasoning, containing 220 what-if questions across STEM, HSS, and Hybrid scenarios. To evaluate free-form responses, we further propose PRISM, which first converts each natural-language explanation into a Response-Derived Semantic Causal Graph of events, states, and mechanisms. On top of this graph, PRISM then jointly applies a Process Metric assessing graph-level causal validity and a Rubric Metric assessing answer-level explanatory adequacy. Evaluating six frontier LLMs with this framework, we find that WhatIfBench remains far from saturated: even the strongest model reaches only a 64.62% final score. Further analysis reveals persistent causal gaps, premise drift, and topology fragmentation, suggesting that fluent counterfactual narratives often mask fragile causal processes. The benchmark, code, and evaluation scripts are available at hrefhttps://github.com/zju-gt/WhatIfBenchWhatIfBench.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Causal reasoning in LLMs, one of 17 directions in LLM reasoning. Builds on 14 of its 32 references that are in the map. First posted on arXiv 28 Aug 2026.

Builds on

14 of 32 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching23 Feb
  2. 2026Uncovering Hidden Correctness in LLM Causal Reasoning via Symbolic Verification29 Jan
  3. 2025Executable Counterfactuals: Improving LLMs' Causal Reasoning Through Code2 Oct
  4. 2025CounterBench: Evaluating and Improving Counterfactual Reasoning in Large Language Models16 Feb
  5. 2024Failure Modes of LLMs for Causal Reasoning on Narratives31 Oct
  6. 2024A Critical Review of Causal Reasoning Benchmarks for Large Language Models10 Jul

Other references · 8

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  3. 2025NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agentscited by 36
  4. 2024Causality for Large Language Modelscited by 25
  5. 2022WikiWhy: Answering and Explaining Cause-and-Effect Questionscited by 21
  6. 2025Can LLMs Reconcile Knowledge Conflicts in Counterfactual Reasoningcited by 7
  7. 2025Flying Pigs, FaR and Beyond: Evaluating LLM Reasoning in Counterfactual Worldscited by 3
  8. 2026Thinking Fast, Thinking Wrong: Intuitiveness Modulates LLM Counterfactual Reasoning in Policy Evaluationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Better Think Thrice: Learning to Reason Causally with Double Counterfactual Consistency
  2. 2025On the Eligibility of LLMs for Counterfactual Reasoning: A Decompositional Study
  3. 2026Deterministic Event-Graph Substrates as World Models for Counterfactual Reasoning
  4. 2024Reasoning Elicitation in Language Models via Counterfactual Feedback
  5. 2025A Non-Interventionist Approach to Causal Reasoning based on Lewisian Counterfactuals
  6. 2026Diagnosing and Mitigating Sycophancy and Skepticism in LLM Causal Judgment
  7. 2026Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoning
  8. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causality