mlsanityData to 7 Oct 2026

Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations

arXiv 2310.11207 ↗First posted 17 Oct 2023cs.CL cs.LGcited by 90

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) such as ChatGPT have demonstrated superior performance on a variety of natural language processing (NLP) tasks including sentiment analysis, mathematical reasoning and summarization. Furthermore, since these models are instruction-tuned on human conversations to produce "helpful" responses, they can and often will produce explanations along with the response, which we call self-explanations. For example, when analyzing the sentiment of a movie review, the model may output not only the positivity of the sentiment, but also an explanation (e.g., by listing the sentiment-laden words such as "fantastic" and "memorable" in the review). How good are these automatically generated self-explanations? In this paper, we investigate this question on the task of sentiment analysis and for feature attribution explanation, one of the most commonly studied settings in the interpretability literature (for pre-ChatGPT models). Specifically, we study different ways to elicit the self-explanations, evaluate their faithfulness on a set of evaluation metrics, and compare them to traditional explanation methods such as occlusion or LIME saliency maps. Through an extensive set of experiments, we find that ChatGPT's self-explanations perform on par with traditional ones, but are quite different from them according to various agreement metrics, meanwhile being much cheaper to produce (as they are generated along with the prediction). In addition, we identified several interesting characteristics of them, which prompt us to rethink many current model interpretability practices in the era of ChatGPT(-like) LLMs.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 41 of its 53 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

41 of 53 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  8. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  11. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  12. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
29 more
  1. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  2. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,086
  3. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  4. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  5. 2017Interpretation of Neural Networks is Fragilecited by 443
  6. 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 440
  7. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  8. 2019Is Attention Interpretable?cited by 387
  9. 2018AI safety via debatecited by 384
  10. 2018On the Robustness of Interpretability Methodscited by 310
  11. 2023Mathematical Capabilities of ChatGPTcited by 263
  12. 2017Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictionscited by 234
  13. 2016Understanding Neural Networks through Representation Erasurecited by 223
  14. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  15. 2022The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspectivecited by 156
  16. 2020Debugging Tests for Model Explanationscited by 146
  17. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  18. 2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
  19. 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
  20. 2019Benchmarking Attribution Methods with Relative Feature Importancecited by 89
  21. 2023Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanationscited by 85
  22. 2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
  23. 2022Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlationcited by 59
  24. 2022Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanationscited by 40
  25. 2021Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classificationcited by 29
  26. 2021The Irrationality of Neural Rationale Modelscited by 16
  27. 2022ExSum: From Local Explanations to Model Understandingcited by 15
  28. 2022The Solvability of Interpretability Evaluation Metricscited by 11
  29. 2021Double Trouble: How to not explain a text classifier's decisions using counterfactuals synthesized by masked language models?cited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 38; 2025: 30; 2026: 18.

90 papers.

Sort:

By area:

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  2. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 107
  3. 2024Are self-explanations from Large Language Models faithful?cited by 101
  4. 2023On Measuring Faithfulness or Self-consistency of Natural Language Explanationscited by 71
  5. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systemscited by 58
  6. 2024Determinants of LLM-assisted Decision-Makingcited by 55
  7. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  8. 2024LC-LLM: Explainable Lane-Change Intention and Trajectory Predictions with Large Language Modelscited by 35
  9. 2024Towards Explainable Traffic Flow Prediction with Large Language Modelscited by 20
  10. 2024Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Modelscited by 19
  11. 2025VeriPlan: Integrating Formal Verification and LLMs into End-User Planningcited by 19
  12. 2024FaithLM: Towards Faithful Explanations for Large Language Modelscited by 18
38 more
  1. 2025V2X-LLM: Enhancing V2X Integration and Understanding in Connected Vehicle Corridorscited by 13
  2. 2024Adversarial Representation Engineering: A General Model Editing Framework for Large Language Modelscited by 12
  3. 2024Evaluating Human Alignment and Model Faithfulness of LLM Rationalecited by 12
  4. 2024Multi-Level Explanations for Generative Language Modelscited by 11
  5. 2024Interpretability Needs a New Paradigmcited by 10
  6. 2024CheckEmbed: Effective Verification of LLM Solutions to Open-Ended Taskscited by 10
  7. 2024The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotatorscited by 10
  8. 2024Generative AI for Requirements Engineering: A Systematic Literature Reviewcited by 10
  9. 2025A Comprehensive Analysis of Large Language Model Outputs: Similarity, Diversity, and Biascited by 10
  10. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  11. 2023Characterizing Large Language Models as Rationalizers of Knowledge-intensive Taskscited by 9
  12. 2024Why Would You Suggest That? Human Trust in Language Model Responsescited by 9
  13. 2024Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Modelscited by 9
  14. 2025From Text to Trust: Empowering AI-assisted Decision Making with Adaptive LLM-powered Analysiscited by 9
  15. 2025Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Frameworkcited by 9
  16. 2024Self-AMPLIFY: Improving Small Language Models with Self Post Hoc Explanationscited by 8
  17. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  18. 2024Shortcut Learning in In-Context Learning: A Surveycited by 8
  19. 2024A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLIcited by 8
  20. 2025Counterfactual Simulatability of LLM Explanations for Generation Taskscited by 7
  21. 2025AlgoTrace: Algorithmic Primitives and Compositional Geometry of Reasoning in Language Modelscited by 7
  22. 2023Image and Data Mining in Reticular Chemistry Using GPT-4Vcited by 6
  23. 2023XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMscited by 5
  24. 2024ProSLM : A Prolog Synergized Language Model for explainable Domain Specific Knowledge Based Question Answeringcited by 5
  25. 2025Exploring Multimodal Prompt for Visualization Authoring with Large Language Modelscited by 5
  26. 2026Can Large Language Models Still Explain Themselves? Investigating the Impact of Quantization on Self-Explanationscited by 5
  27. 2024Free-text Rationale Generation under Readability Level Controlcited by 4
  28. 2025Reasoning-Grounded Natural Language Explanations for Language Modelscited by 4
  29. 2025Advancing AI-Scientist Understanding: Multi-Agent LLMs with Interpretable Physics Reasoningcited by 4
  30. 2025Modular Machine Learning: An Indispensable Path towards New-Generation Large Language Modelscited by 4
  31. 2025HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Drivingcited by 4
  32. 2025LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inferencecited by 4
  33. 2024Leveraging Large Language Models for Concept Graph Recovery and Question Answering in NLP Educationcited by 3
  34. 2024Leveraging the Power of LLMs: A Fine-Tuning Approach for High-Quality Aspect-Based Summarizationcited by 3
  35. 2024DILA: Dictionary Label Attention for Mechanistic Interpretability in High-dimensional Multi-label Medical Coding Predictioncited by 3
  36. 2024The Impact of Generative Artificial Intelligence on Ideation and the performance of Innovation Teams (Preprint)cited by 3
  37. 2025Can AI Examine Novelty of Patents?: Novelty Evaluation Based on the Correspondence between Patent Claim and Prior Artcited by 3
  38. 2025PEDANTIC: A Dataset for the Automatic Examination of Definiteness in Patent Claimscited by 3

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy
  2. 2025Can LLMs Explain Themselves Counterfactually?
  3. 2024From Feature Importance to Natural Language Explanations Using LLMs with RAG
  4. 2025Self-Critique and Refinement for Faithful Natural Language Explanations
  5. 2024Rethinking Interpretability in the Era of Large Language Models
  6. 2024Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models
  7. 2026Do LLM Self-Explanations Help Users Predict Model Behavior? Evaluating Counterfactual Simulatability with Pragmatic Perturbations
  8. 2023Self-training Strategies for Sentiment Analysis: An Empirical Study