mlsanityData to 7 Oct 2026

Interventional Grounding Audits: Black-Box Premise-Dependency Tests for LLM Chain-of-Thought via Predicate Substitution

arXiv 2607.13069 ↗First posted 11 Jul 2026cs.AI cs.CL cs.LOcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Large language models produce chain-of-thought (CoT) reasoning that appears logically sound yet may not genuinely depend on its stated premises. We introduce interventional grounding audits, a black-box, step-level test of premise dependency: we intervene on a single premise by substituting its target predicate with a fresh symbol, re-run the model, and check whether each reasoning step's normalized conclusion (canonical predicate form) changes. We evaluate on ProntoQA, a synthetic multi-hop deductive reasoning benchmark with gold proof trees, where step-level premise dependencies are known. Applied to 50 ProntoQA problems with GPT-4o, our method achieves F1 = 0.806 on detecting proof-tree dependencies (F1 = 0.885 on predicate-determining dependencies; Recall = 100%), significantly outperforming a self-consistency baseline (F1 = 0.343; 95% bootstrap CIs non-overlapping). We further identify that 66% of correctly-solved problems contain at least one aligned step insensitive to a direct proof-tree dependency under consistent substitution -- all involving entity-introduction premises, a documented blind spot of the consistent-substitution evaluator -- a "right answer, wrong reasoning" signal invisible to passive methods. All audit certificates, raw outputs, and reproduction scripts are available in a public GitHub repository, and we discuss scope limits beyond formal, parsable benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 6 of its 12 references that are in the map. First posted on arXiv 11 Jul 2026.

Builds on

6 of 12 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting7 May
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 3

  1. 2021Causal Abstractions of Neural Networkscited by 401
  2. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  3. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Tell-Tale Trace: Detecting Reasoning Failures in LLMs Using Chain-of-Thought Dynamics
  2. 2026C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning
  3. 2026Mechanistic Evidence for Faithfulness Decay in Chain-of-Thought Reasoning
  4. 2026LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs
  5. 2025FRIT: Using Causal Importance to Improve Chain-of-Thought Faithfulness
  6. 2024Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs
  7. 2025FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning
  8. 2026Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations