# Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations

arXiv: https://arxiv.org/abs/2504.14150 · HTML: https://mlsanity.com/paper/2504.14150/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 11 references that are in the map.
- First posted on arXiv 19 Apr 2025.

## Builds on (most cited first, 9 of 9)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC](https://mlsanity.com/paper/1507.04544/) (2015)
- [Explaining Predictions of Non-Linear Classifiers in NLP](https://mlsanity.com/paper/1606.07298/) (2016)

## Built on by (most cited first, 30 of 45)

- [FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning](https://mlsanity.com/paper/2510.04040/) (2025)
- [How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding](https://mlsanity.com/paper/2507.22928/) (2025)
- [Counterfactual Simulation Training for Chain-of-Thought Faithfulness](https://mlsanity.com/paper/2602.20710/) (2026)
- [GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenes](https://mlsanity.com/paper/2511.22645/) (2025)
- [A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behavior](https://mlsanity.com/paper/2602.02639/) (2026)
- [A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments](https://mlsanity.com/paper/2509.25609/) (2025)
- [RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models](https://mlsanity.com/paper/2602.17053/) (2026)
- [Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2510.13975/) (2025)
- [Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations](https://mlsanity.com/paper/2510.11196/) (2025)
- [Faithful-First Reasoning, Planning, and Acting for Multimodal LLMs](https://mlsanity.com/paper/2511.08409/) (2025)
- [Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning](https://mlsanity.com/paper/2604.22074/) (2026)
- [Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.13272/) (2025)
- [Towards Transparent Reasoning: What Drives Faithfulness in Large Language Models?](https://mlsanity.com/paper/2510.24236/) (2025)
- [Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilities](https://mlsanity.com/paper/2603.10396/) (2026)
- [Detecting and Suppressing Reward Hacking with Gradient Fingerprints](https://mlsanity.com/paper/2604.16242/) (2026)
- [Leveraging Imperfection with MEDLEY A Multi-Model Approach Harnessing Bias in Medical AI](https://mlsanity.com/paper/2508.21648/) (2025)
- [Unspoken Hints: Accuracy Without Acknowledgement in LLM Reasoning](https://mlsanity.com/paper/2509.26041/) (2025)
- [Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agents](https://mlsanity.com/paper/2510.02204/) (2025)
- [Epidemiology of Large Language Models: A Benchmark for Observational Distribution Knowledge](https://mlsanity.com/paper/2511.03070/) (2025)
- [Detecting Unfaithful Chain-of-Thought via Circuit-Guided Internal-External Discrepancy](https://mlsanity.com/paper/2605.25603/) (2026)
- [Argument-Based Consistency in Toxicity Explanations of LLMs](https://mlsanity.com/paper/2506.19113/) (2025)
- [Rewrite-to-Rank: Optimizing Ad Visibility via Retrieval-Aware Text Rewriting](https://mlsanity.com/paper/2507.21099/) (2025)
- [Is General-Purpose AI Reasoning Sensitive to Data-Induced Cognitive Biases? Dynamic Benchmarking on Typical Software Engineering Dilemmas](https://mlsanity.com/paper/2508.11278/) (2025)
- [Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations](https://mlsanity.com/paper/2510.17256/) (2025)
- [Ensembling LLM-Induced Decision Trees for Explainable and Robust Error Detection](https://mlsanity.com/paper/2512.07246/) (2025)
- [When AI Persuades: Adversarial Explanation Attacks on Human Trust in AI-Assisted Decision Making](https://mlsanity.com/paper/2602.04003/) (2026)
- [Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning](https://mlsanity.com/paper/2603.13988/) (2026)
- [When Persona Attributes Improve Population Alignment in Large Language Models](https://mlsanity.com/paper/2609.02526/) (2026)
- [Investigating Faithfulness in Large Audio Language Models](https://mlsanity.com/paper/2509.22363/) (2025)
- [Do Vision-Language-Action Models Mean What They Say? On the Role of Faithfulness in Embodied Reasoning](https://mlsanity.com/paper/2607.04681/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are capable of generating plausible explanations of how they arrived at an answer to a question. However, these explanations can misrepresent the model's "reasoning" process, i.e., they can be unfaithful. This, in turn, can lead to over-trust and misuse. We introduce a new approach for measuring the faithfulness of LLM explanations. First, we provide a rigorous definition of faithfulness. Since LLM explanations mimic human explanations, they often reference high-level concepts in the input question that purportedly influenced the model. We define faithfulness in terms of the difference between the set of concepts that LLM explanations imply are influential and the set that truly are. Second, we present a novel method for estimating faithfulness that is based on: (1) using an auxiliary LLM to modify the values of concepts within model inputs to create realistic counterfactuals, and (2) using a Bayesian hierarchical model to quantify the causal effects of concepts at both the example- and dataset-level. Our experiments show that our method can be used to quantify and discover interpretable patterns of unfaithfulness. On a social bias task, we uncover cases where LLM explanations hide the influence of social bias. On a medical question answering task, we uncover cases where LLM explanations provide misleading claims about which pieces of evidence influenced the model's decisions.