# How Interpretable are Reasoning Explanations from Prompting Large Language Models?

arXiv: https://arxiv.org/abs/2402.11863 · HTML: https://mlsanity.com/paper/2402.11863/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 35 of its 45 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 30 of 35)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [Faithful Chain-of-Thought Reasoning](https://mlsanity.com/paper/2301.13379/) (2023)
- [Rationalizing Neural Predictions](https://mlsanity.com/paper/1606.04155/) (2016)
- [Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies](https://mlsanity.com/paper/2308.03188/) (2023)
- [Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future](https://mlsanity.com/paper/2309.15402/) (2023)
- [ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning](https://mlsanity.com/paper/2212.07919/) (2022)
- [Measuring Association Between Labels and Free-Text Rationales](https://mlsanity.com/paper/2010.12762/) (2020)
- [WT5?! Training Text-to-Text Models to Explain their Predictions](https://mlsanity.com/paper/2004.14546/) (2020)
- [Rethinking with Retrieval: Faithful Large Language Model Inference](https://mlsanity.com/paper/2301.00303/) (2022)
- [Faithful Reasoning Using Large Language Models](https://mlsanity.com/paper/2208.14271/) (2022)
- [Question Decomposition Improves the Faithfulness of Model-Generated Reasoning](https://mlsanity.com/paper/2307.11768/) (2023)
- [Faithfulness Tests for Natural Language Explanations](https://mlsanity.com/paper/2305.18029/) (2023)
- [Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answering](https://mlsanity.com/paper/2308.13259/) (2023)
- [NILE : Natural Language Inference with Faithful Natural Language Explanations](https://mlsanity.com/paper/2005.12116/) (2020)
- [Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?](https://mlsanity.com/paper/2010.04119/) (2020)
- [ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness](https://mlsanity.com/paper/2304.10703/) (2023)
- [PINTO: Faithful Language Reasoning Using Prompt-Generated Rationales](https://mlsanity.com/paper/2211.01562/) (2022)
- [Can Rationalization Improve Robustness?](https://mlsanity.com/paper/2204.11790/) (2022)

## Built on by (most cited first, 29 of 29)

- [Chain of Thoughtlessness? An Analysis of CoT in Planning](https://mlsanity.com/paper/2405.04776/) (2024)
- [Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models](https://mlsanity.com/paper/2405.13872/) (2024)
- [Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies](https://mlsanity.com/paper/2502.08554/) (2025)
- [Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training](https://mlsanity.com/paper/2509.03403/) (2025)
- [FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning](https://mlsanity.com/paper/2510.04040/) (2025)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspective](https://mlsanity.com/paper/2504.03255/) (2025)
- [How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding](https://mlsanity.com/paper/2508.20279/) (2025)
- [Explainable Natural Language Processing for Corporate Sustainability Analysis](https://mlsanity.com/paper/2407.17487/) (2024)
- [Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations](https://mlsanity.com/paper/2504.05294/) (2025)
- [Clinical Insights: A Comprehensive Review of Language Models in Medicine](https://mlsanity.com/paper/2408.11735/) (2024)
- [What Level of Automation is "Good Enough"? A Benchmark of Large Language Models for Meta-Analysis Data Extraction](https://mlsanity.com/paper/2507.15152/) (2025)
- [Do explanations generalize across large reasoning models?](https://mlsanity.com/paper/2601.11517/) (2026)
- [Towards Transparent AI: A Survey on Explainable Language Models](https://mlsanity.com/paper/2509.21631/) (2025)
- [Unreal Thinking: Chain-of-Thought Hijacking via Two-stage Backdoor](https://mlsanity.com/paper/2604.09235/) (2026)
- [Complexity in Complexity: Understanding Visual Complexity Through Structure, Color, and Surprise](https://mlsanity.com/paper/2501.15890/) (2025)
- [Bonsai: Interpretable Tree-Adaptive Grounded Reasoning](https://mlsanity.com/paper/2504.03640/) (2025)
- [Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics](https://mlsanity.com/paper/2504.17665/) (2025)
- [SynthEHR-Eviction: Enhancing Eviction SDoH Detection with LLM-Augmented Synthetic EHR Data](https://mlsanity.com/paper/2507.07421/) (2025)
- [Protocode: Prototype-Driven Interpretability for Code Generation in LLMs](https://mlsanity.com/paper/2509.25247/) (2025)
- [SVBRD-LLM: Self-Verifying Behavioral Rule Discovery for Autonomous Vehicle Identification](https://mlsanity.com/paper/2511.14977/) (2025)
- [OmniSapiens: A Foundation Model for Social Behavior Processing via Heterogeneity-Aware Relative Policy Optimization](https://mlsanity.com/paper/2602.10635/) (2026)
- [YouTube Comments Decoded: Leveraging LLMs for Low Resource Language Classification](https://mlsanity.com/paper/2411.05039/) (2024)
- [Rationale-Enhanced Decoding for Multi-modal Chain-of-Thought](https://mlsanity.com/paper/2507.07685/) (2025)
- [Can LLM-Generated Textual Explanations Enhance Model Classification Performance? An Empirical Study](https://mlsanity.com/paper/2508.09776/) (2025)
- [Reasoning About Reasoning: Towards Informed and Reflective Use of LLM Reasoning in HCI](https://mlsanity.com/paper/2510.22978/) (2025)
- [Fundamental Limitation in Explaining AI](https://mlsanity.com/paper/2605.24727/) (2026)
- [A Multi-Dimensional Evaluation of Explainability in Media Bias Detection](https://mlsanity.com/paper/2607.19954/) (2026)
- [Enhancing SLMs for Sustainable Code Optimization in Radio-Astronomy](https://mlsanity.com/paper/2607.21677/) (2026)

## Abstract (arXiv)

Prompt Engineering has garnered significant attention for enhancing the performance of large language models across a multitude of tasks. Techniques such as the Chain-of-Thought not only bolster task performance but also delineate a clear trajectory of reasoning steps, offering a tangible form of explanation for the audience. Prior works on interpretability assess the reasoning chains yielded by Chain-of-Thought solely along a singular axis, namely faithfulness. We present a comprehensive and multifaceted evaluation of interpretability, examining not only faithfulness but also robustness and utility across multiple commonsense reasoning benchmarks. Likewise, our investigation is not confined to a single prompting technique; it expansively covers a multitude of prevalent prompting techniques employed in large language models, thereby ensuring a wide-ranging and exhaustive evaluation. In addition, we introduce a simple interpretability alignment technique, termed Self-Entailment-Alignment Chain-of-thought, that yields more than 70% improvements across multiple dimensions of interpretability. Code is available at https://github.com/SenticNet/CoTinterpretability