mlsanityData to 7 Oct 2026

Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps

arXiv 2502.14829 ↗First posted 20 Feb 2025cs.CLcited by 43

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

When prompted to think step-by-step, language models (LMs) produce a chain of thought (CoT), a sequence of reasoning steps that the model supposedly used to produce its prediction. Despite much work on CoT prompting, it is unclear if reasoning verbalized in a CoT is faithful to the models' parametric beliefs. We introduce a framework for measuring parametric faithfulness of generated reasoning, and propose Faithfulness by Unlearning Reasoning steps (FUR), an instance of this framework. FUR erases information contained in reasoning steps from model parameters, and measures faithfulness as the resulting effect on the model's prediction. Our experiments with four LMs and five multi-hop multi-choice question answering (MCQA) datasets show that FUR is frequently able to precisely change the underlying models' prediction for a given instance by unlearning key steps, indicating when a CoT is parametrically faithful. Further analysis shows that CoTs generated by models post-unlearning support different answers, hinting at a deeper effect of unlearning.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 59 of its 83 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

59 of 83 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers4 Feb
  2. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning18 Sep
  3. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step23 May
  4. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Models24 Apr
  5. 2024How Likely Do LLMs with CoT Mimic Human Reasoning?25 Feb
  6. 2024Chain-of-Thought Unfaithfulness as Disguised Accuracy22 Feb
  7. 2024A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains1 Feb
  8. 2023On Measuring Faithfulness or Self-consistency of Natural Language Explanations13 Nov
  9. 2023Measuring Faithfulness in Chain-of-Thought Reasoning17 Jul
  10. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting7 May
  11. 2022Complexity-Based Prompting for Multi-Step Reasoning3 Oct
  12. 2022Large Language Models are Zero-Shot Reasoners24 May
1 more
  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar

Other references · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  12. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
34 more
  1. 2019Machine Unlearningcited by 1,020
  2. 2022Mass-Editing Memory in a Transformercited by 976
  3. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  4. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  5. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  6. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  7. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  8. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
  9. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  10. 2023How Language Model Hallucinations Can Snowballcited by 310
  11. 2020Descent-to-Delete: Gradient-Based Methods for Machine Unlearningcited by 303
  12. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  13. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  14. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  15. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  16. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 160
  17. 2023Faithfulness Tests for Natural Language Explanationscited by 107
  18. 2024Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Modelscited by 104
  19. 2024Are self-explanations from Large Language Models faithful?cited by 101
  20. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  21. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 93
  22. 2023Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanationscited by 85
  23. 2024Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queriescited by 78
  24. 2019Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanationscited by 71
  25. 2023Why Does ChatGPT Fall Short in Providing Truthful Answers?cited by 54
  26. 2022(QA)2: Question Answering with Questionable Assumptionscited by 51
  27. 2021Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answeringcited by 41
  28. 2023Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-Text Rationalescited by 37
  29. 2024Erasing Conceptual Knowledge from Language Modelscited by 34
  30. 2024The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Modelscited by 29
  31. 2024Controllable Context Sensitivity and the Knob Behind Itcited by 22
  32. 2024Distinguishing Ignorance from Error in LLM Hallucinationscited by 14
  33. 2024REVS: Unlearning Sensitive Information in Language Models via Rank Editing in the Vocabulary Spacecited by 12
  34. 2024To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguitycited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 30.

43 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  3. 2025FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoningcited by 25
  4. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  5. 2025Verifying Chain-of-Thought Reasoning via Its Computational Graphcited by 20
  6. 2026CausalArmor: Efficient Indirect Prompt Injection Guardrails via Causal Attributioncited by 13
  7. 2025A Causal Lens for Evaluating Faithfulness Metricscited by 12
  8. 2025A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Modelscited by 12
  9. 2026A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behaviorcited by 12
  10. 2025Do Cognitively Interpretable Reasoning Traces Improve LLM Performance?cited by 11
  11. 2026Dynamics Within Latent Chain-of-Thought: An Empirical Study of Causal Structurecited by 9
  12. 2025Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillationcited by 8
31 more
  1. 2025Analysing Chain of Thought Dynamics: Active Guidance or Unfaithful Post-hoc Rationalisation?cited by 8
  2. 2026Can Large Language Models Still Explain Themselves? Investigating the Impact of Quantization on Self-Explanationscited by 5
  3. 2026Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structurescited by 5
  4. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  5. 2026Reasoning Models Will Sometimes Lie About Their Reasoningcited by 4
  6. 2026Detecting and Suppressing Reward Hacking with Gradient Fingerprintscited by 4
  7. 2026Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truthcited by 3
  8. 2026Diagnosing Pathological Chain-of-Thought in Reasoning Modelscited by 2
  9. 2026When Reasoning Traces Become Performative: Step-Level Evidence that Chain-of-Thought Is an Imperfect Oversight Channelcited by 2
  10. 2026Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoningcited by 2
  11. 2026To Isolate or to Score? Model-Adaptive Assessment for Cost-Efficient Multi-Agent RAGcited by 2
  12. 2026Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compressioncited by 2
  13. 2025Do LLMs Know They Are Being Tested? Evaluation Awareness and Incentive-Sensitive Failures in GPT-OSS-20Bcited by 1
  14. 2025Support-Contra Asymmetry in LLM Explanationscited by 1
  15. 2026Measuring and curing reasoning rigidity: from decorative chain-of-thought to genuine faithfulnesscited by 1
  16. 2026Reasoning Models Know What's Important, and Encode It in Their Activationscited by 1
  17. 2026Compliance vs. Sensibility: On the Reasoning Controllability in Large Language Modelscited by 1
  18. 2026Instructions Shape Production of Language, not Processingcited by 1
  19. 2026ConsisGuard: Aligning Safety Deliberation with Policy Enforcement in LLM Guardrailscited by 1
  20. 2026Cosine Misleads: Auxiliary Losses Reshape Vision Language Models, Not Their Latentscited by 1
  21. 2026Decoding Hidden Deception in Reasoning LLMs: Activation Explainers for Deception Auditingcited by 1
  22. 2026Evaluation of Multi-Turn Consistency in LLM Agents: Survival Analysis and Failure-Rationale Taxonomycited by 1
  23. 2025NeuroFaith: Evaluating LLM Self-Explanation Faithfulness via Internal Representation Alignmentcited by 0
  24. 2026TRUE: A Trustworthy Unified Explanation Framework for Large Language Model Reasoningcited by 0
  25. 2026Agentic Interpretation: Lattice-Structured Evidence for LLM-Based Program Analysiscited by 0
  26. 2026Sycophancy Towards Researchers Drives Performative Misalignmentcited by 0
  27. 2026Context-Masked Truncated Reasoning Audits for Answer-Key Dependence in LLM Tutorscited by 0
  28. 2026Fusion Training for Mathematical Generalization in Large Language Modelscited by 0
  29. 2026A primer on evaluation methods for large language models in healthcarecited by 0
  30. 2026Taming CoT Obfuscation in VLMs: From Mechanistic Evidence to Activation Enforcementcited by 0
  31. 2026What Do Rationales Communicate? A Message-Intervention Study in Role-Specialized QAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mechanistic Evidence for Faithfulness Decay in Chain-of-Thought Reasoning
  2. 2026Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?
  3. 2026Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignment
  4. 2025A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages
  5. 2023Faithful Chain-of-Thought Reasoning
  6. 2025How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding
  7. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithful
  8. 2025Reasoning Relay: Evaluating Stability and Interchangeability of Large Language Models in Mathematical Reasoning