# Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models

arXiv: https://arxiv.org/abs/2402.04614 · HTML: https://mlsanity.com/paper/2402.04614/

## Key facts

- Cited by 105 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 7 Feb 2024.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [Visualizing and Understanding Neural Models in NLP](https://mlsanity.com/paper/1506.01066/) (2015)
- [Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing](https://mlsanity.com/paper/2102.12060/) (2021)
- [AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap](https://mlsanity.com/paper/2306.01941/) (2023)
- [Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT](https://mlsanity.com/paper/2004.14786/) (2020)
- [The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models](https://mlsanity.com/paper/2401.05618/) (2024)
- [Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrong](https://mlsanity.com/paper/2310.12558/) (2023)
- [LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond](https://mlsanity.com/paper/2305.14540/) (2023)
- [LLM Harmony: Multi-Agent Communication for Problem Solving](https://mlsanity.com/paper/2401.01312/) (2024)
- [Why Can Large Language Models Generate Correct Chain-of-Thoughts?](https://mlsanity.com/paper/2310.13571/) (2023)
- [OPT-R: Exploring the Role of Explanations in Finetuning and Prompting for Reasoning Skills of Large Language Models](https://mlsanity.com/paper/2305.12001/) (2023)
- [DCR: Divide-and-Conquer Reasoning for Multi-choice Question Answering with LLMs](https://mlsanity.com/paper/2401.05190/) (2024)

## Built on by (most cited first, 30 of 105)

- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyond](https://mlsanity.com/paper/2503.21614/) (2025)
- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [Are DeepSeek R1 And Other Reasoning Models More Faithful?](https://mlsanity.com/paper/2501.08156/) (2025)
- [Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above](https://mlsanity.com/paper/2502.14127/) (2025)
- [Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?](https://mlsanity.com/paper/2501.15463/) (2025)
- [Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps](https://mlsanity.com/paper/2502.14829/) (2025)
- [On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models](https://mlsanity.com/paper/2406.10625/) (2024)
- [Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought](https://mlsanity.com/paper/2603.05488/) (2026)
- [Neither Valid nor Reliable? Investigating the Use of LLMs as Judges](https://mlsanity.com/paper/2508.18076/) (2025)
- [FakeBench: Probing Explainable Fake Image Detection via Large Multimodal Models](https://mlsanity.com/paper/2404.13306/) (2024)
- [LLMs generate structurally realistic social networks but overestimate political homophily](https://mlsanity.com/paper/2408.16629/) (2024)
- [On the Impact of Fine-Tuning on Chain-of-Thought Reasoning](https://mlsanity.com/paper/2411.15382/) (2024)
- [Energy-Based Transformers are Scalable Learners and Thinkers](https://mlsanity.com/paper/2507.02092/) (2025)
- [A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models](https://mlsanity.com/paper/2509.03871/) (2025)
- [Measuring the Faithfulness of Thinking Drafts in Large Reasoning Models](https://mlsanity.com/paper/2505.13774/) (2025)
- [Transformer Circuit Faithfulness Metrics are not Robust](https://mlsanity.com/paper/2407.08734/) (2024)
- [LLM-Generated Black-box Explanations Can Be Adversarially Helpful](https://mlsanity.com/paper/2405.06800/) (2024)
- [The Fusion of Large Language Models and Formal Methods for Trustworthy AI Agents: A Roadmap](https://mlsanity.com/paper/2412.06512/) (2024)
- [How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding](https://mlsanity.com/paper/2507.22928/) (2025)
- [Interpretable LLM-based Table Question Answering](https://mlsanity.com/paper/2412.12386/) (2024)
- [Correctness is not Faithfulness in RAG Attributions](https://mlsanity.com/paper/2412.18004/) (2024)
- [On the Faithfulness of Visual Thinking: Measurement and Enhancement](https://mlsanity.com/paper/2510.23482/) (2025)
- [Towards Faithful Natural Language Explanations: A Study Using Activation Patching in Large Language Models](https://mlsanity.com/paper/2410.14155/) (2024)
- [Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing](https://mlsanity.com/paper/2507.10587/) (2025)
- [Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning](https://mlsanity.com/paper/2402.14856/) (2024)
- [Probabilistic Soundness Guarantees in LLM Reasoning Chains](https://mlsanity.com/paper/2507.12948/) (2025)
- [A Pragmatic Way to Measure Chain-of-Thought Monitorability](https://mlsanity.com/paper/2510.23966/) (2025)
- [Evaluating Human Alignment and Model Faithfulness of LLM Rationale](https://mlsanity.com/paper/2407.00219/) (2024)
- [Traffic-R1: Reinforced LLMs Bring Human-Like Reasoning to Traffic Signal Control Systems](https://mlsanity.com/paper/2508.02344/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) are deployed as powerful tools for several natural language processing (NLP) applications. Recent works show that modern LLMs can generate self-explanations (SEs), which elicit their intermediate reasoning steps for explaining their behavior. Self-explanations have seen widespread adoption owing to their conversational and plausible nature. However, there is little to no understanding of their faithfulness. In this work, we discuss the dichotomy between faithfulness and plausibility in SEs generated by LLMs. We argue that while LLMs are adept at generating plausible explanations -- seemingly logical and coherent to human users -- these explanations do not necessarily align with the reasoning processes of the LLMs, raising concerns about their faithfulness. We highlight that the current trend towards increasing the plausibility of explanations, primarily driven by the demand for user-friendly interfaces, may come at the cost of diminishing their faithfulness. We assert that the faithfulness of explanations is critical in LLMs employed for high-stakes decision-making. Moreover, we emphasize the need for a systematic characterization of faithfulness-plausibility requirements of different real-world applications and ensure explanations meet those needs. While there are several approaches to improving plausibility, improving faithfulness is an open challenge. We call upon the community to develop novel methods to enhance the faithfulness of self explanations thereby enabling transparent deployment of LLMs in diverse high-stakes settings.