mlsanityData to 7 Oct 2026

Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluation

arXiv 2603.20172 ↗First posted 20 Mar 2026cs.CL cs.AI cs.LGcited by 4

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Recent work on chain-of-thought (CoT) faithfulness reports single aggregate numbers (e.g., DeepSeek-R1 acknowledges hints 39% of the time), implying that faithfulness is an objective, measurable property of a model. This paper provides evidence that it is not. Three classifiers (a regex-only detector, a regex-plus-LLM pipeline, and a Claude Sonnet 4 judge) are applied to 10,276 influenced reasoning traces from 12 open-weight models spanning 9 families and 7B to 1T parameters. On identical data, these classifiers produce faithfulness rates of 74.4%, 82.6%, and 69.7%. Per-model gaps range from 2.6 to 30.6 percentage points; all pairwise McNemar tests are significant (p < 0.001). The disagreements are systematic: Cohen's kappa ranges from 0.06 ("slight") for sycophancy hints to 0.42 ("moderate") for grader hints, and the asymmetry is pronounced: for sycophancy, 883 cases are classified as faithful by the pipeline but unfaithful by the Sonnet judge, while only 2 go the other direction. Classifier choice can also reverse model rankings: Qwen3.5-27B ranks 1st under the pipeline but 7th under Sonnet; OLMo-3.1-32B moves from 9th to 3rd. Different classifiers operationalize faithfulness at different levels of stringency (lexical mention versus epistemic dependence), yielding divergent measurements on the same behavior. These results indicate that published faithfulness numbers cannot be meaningfully compared across studies using different classifiers, and that future evaluations should report sensitivity ranges across multiple classification methodologies.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 43 references that are in the map. First posted on arXiv 20 Mar 2026.

Builds on

34 of 43 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026Mechanistic Evidence for Faithfulness Decay in Chain-of-Thought Reasoning4 Feb
  2. 2025Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization28 Dec
  3. 2025Investigating CoT Monitorability in Large Reasoning Models11 Nov
  4. 2025Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbosity31 Oct
  5. 2025FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning5 Oct
  6. 2025Reasoning Models Don't Always Say What They Think8 May
  7. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithful11 Mar
  8. 2024On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models15 Jun
  9. 2024Dissociation of Faithful and Unfaithful Reasoning in LLMs23 May
  10. 2023On Measuring Faithfulness or Self-consistency of Natural Language Explanations13 Nov
  11. 2023Preventing Language Models From Hiding Their Reasoning27 Oct
  12. 2023Measuring Faithfulness in Chain-of-Thought Reasoning17 Jul
3 more
  1. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting7 May
  2. 2023Faithful Chain-of-Thought Reasoning31 Jan
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 19

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  6. 2024A Survey on LLM-as-a-Judgecited by 1,384
  7. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  8. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  9. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  10. 2024Alignment faking in large language modelscited by 316
  11. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 154
  12. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 107
7 more
  1. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  2. 2025Olmo 3cited by 95
  3. 2025Are DeepSeek R1 And Other Reasoning Models More Faithful?cited by 69
  4. 2025Auditing language models for hidden objectivescited by 68
  5. 2025Measuring the Faithfulness of Thinking Drafts in Large Reasoning Modelscited by 22
  6. 2025A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Modelscited by 12
  7. 2025MONICA: Real-Time Monitoring and Calibration of Chain-of-Thought Sycophancy in Large Reasoning Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?cited by 8
  2. 2026Why Models Know But Don't Say: Chain-of-Thought Faithfulness Divergence Between Thinking Tokens and Answers in Open-Weight Reasoning Modelscited by 4
  3. 2026Investigating the Interplay between Contextual and Parametric Chain-of-Thought Faithfulness under Optimizationcited by 2
  4. 2026The Compliance Trap: How Structural Constraints Degrade Frontier AI Metacognition Under Adversarial Pressurecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truth
  2. 2024Chain-of-Thought Unfaithfulness as Disguised Accuracy
  3. 2025A Causal Lens for Evaluating Faithfulness Metrics
  4. 2026C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning
  5. 2026Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning
  6. 2026SIFT: Robust Meta-Faithfulness Verification of Chain-of-Thought Reasoning Under Distribution Shift
  7. 2024Are self-explanations from Large Language Models faithful?
  8. 2026Do Models Know Why They Changed Their Mind? Interpretability and Faithfulness of Chain-of-Thought Under Knowledge Conflict