# Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness

arXiv: https://arxiv.org/abs/2610.12361 · HTML: https://mlsanity.com/paper/2610.12361/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 11 of its 25 references that are in the map.
- First posted on arXiv 8 Oct 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 11 of 11)

- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [LegalDelta: Enhancing Legal Reasoning in LLMs via Reinforcement Learning with Chain-of-Thought Guided Information Gain](https://mlsanity.com/paper/2508.12281/) (2025)
- [When Agents Commit Too Soon: Diagnosing Premature Commitment in LLM Agents](https://mlsanity.com/paper/2606.22936/) (2026)
- [Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures](https://mlsanity.com/paper/2603.16475/) (2026)
- [Beyond the Commitment Boundary: Probing Epiphenomenal Chain-of-Thought in Large Reasoning Models](https://mlsanity.com/paper/2606.13603/) (2026)
- [LegalCiteBench: Evaluating Citation Reliability in Legal Language Models](https://mlsanity.com/paper/2605.10186/) (2026)
- [Know Your Limits : On the Faithfulness of LLMs as Solvers and Autoformalizers in Legal Reasoning](https://mlsanity.com/paper/2606.16118/) (2026)
- [When Does a Language Model Commit? A Finite-Answer Theory of Pre-Verbalization Commitment](https://mlsanity.com/paper/2605.06723/) (2026)
- [Who Checks the Citations? Benchmarking Legal Hallucination Detection](https://mlsanity.com/paper/2606.21155/) (2026)
- [Citation Grounding Measures the Oracle: Graph Coverage Determines Reported LLM Hallucination Rates in Law](https://mlsanity.com/paper/2606.00898/) (2026)

## Abstract (arXiv)

Large language models increasingly justify legal decisions by naming the statute or precedent behind a verdict, treated as evidence that the decision follows from it. We test this directly: holding case facts fixed, we substitute the named legal authority for an unrelated one and decode a model's evolving verdict from its hidden states. Across seven open-weight models (8B-70B) and four benchmarks spanning judicial and contractual reasoning, when explicitly required to justify a verdict by naming the governing authority, models name the correct one in 66.7%-100% of generations, while the verdict changing when the authority changes is far less consistent: 0.0%-21.7% on CaseHOLD, 30.0%-76.7% on ECHR and SCOTUS, and 43.3%-50.0% on ContractNLI. Neither scale nor a purpose-built legal-reasoning model (a best-effort LoRA reproduction; Section 6) closes this gap. A red-teaming evaluation on five core models finds compliance with an adversarial instruction hidden in the case facts (73.3%-96.4%) exceeds verdict-swap sensitivity by a wide margin, holding without exception across model rankings. Naming a legal authority is thus a poor proxy for a verdict's dependence on it, while the same verdict remains separately vulnerable to adversarial manipulation. Both findings replicate across checks ruling out prompt-wording noise and confounded sampling, and bear directly on the use of generated legal explanations as compliance or audit artefacts.