mlsanityData to 7 Oct 2026

Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations

arXiv 2307.08678 ↗First posted 17 Jul 2023cs.CL cs.AI cs.LGcited by 85

Not on the map. Why?

Cited by 23 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) are trained to imitate humans to explain human decisions. However, do LLMs explain themselves? Can they help humans build mental models of how LLMs process different inputs? To answer these questions, we propose to evaluate counterfactual simulatability of natural language explanations: whether an explanation can enable humans to precisely infer the model's outputs on diverse counterfactuals of the explained input. For example, if a model answers "yes" to the input question "Can eagles fly?" with the explanation "all birds can fly", then humans would infer from the explanation that it would also answer "yes" to the counterfactual input "Can penguins fly?". If the explanation is precise, then the model's answer should match humans' expectations. We implemented two metrics based on counterfactual simulatability: precision and generality. We generated diverse counterfactuals automatically using LLMs. We then used these metrics to evaluate state-of-the-art LLMs (e.g., GPT-4) on two tasks: multi-hop factual reasoning and reward modeling. We found that LLM's explanations have low precision and that precision does not correlate with plausibility. Therefore, naively optimizing human approvals (e.g., RLHF) may not be a sufficient solution.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 45 of its 70 references that are in the map. First posted on arXiv 17 Jul 2023.

Builds on

45 of 70 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  11. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  12. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
33 more
  1. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  2. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  3. 2020Aligning AI With Shared Human Valuescited by 706
  4. 2022Large Language Models Can Self-Improvecited by 656
  5. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  6. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  7. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  8. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  9. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  10. 2019Counterfactual Visual Explanationscited by 312
  11. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  12. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  13. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 189
  14. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  15. 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
  16. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  17. 2022Faithful Reasoning Using Large Language Modelscited by 130
  18. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  19. 2020NILE : Natural Language Inference with Faithful Natural Language Explanationscited by 96
  20. 2020Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?cited by 95
  21. 2022Interpreting Language Models with Contrastive Explanationscited by 91
  22. 2021Contrastive Explanations for Model Interpretabilitycited by 75
  23. 2018How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanationcited by 71
  24. 2018Faithful Multimodal Explanation for Visual Question Answeringcited by 64
  25. 2018Contrastive Explanation: A Structural-Model Approachcited by 64
  26. 2017The Promise and Peril of Human Evaluation for Model Interpretabilitycited by 52
  27. 2018Do Explanations make VQA Models more Predictable to a Human?cited by 51
  28. 2023Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-Text Rationalescited by 37
  29. 2018Teaching Categories to Human Learners with Visual Explanationscited by 30
  30. 2022REV: Information-Theoretic Evaluation of Free-Text Rationalescited by 22
  31. 2021Can Explanations Be Useful for Calibrating Black Box Models?cited by 20
  32. 2020Evaluating Explanation Methods for Neural Machine Translationcited by 14
  33. 2022Logical Satisfiability of Counterfactuals for Faithful Explanations in NLIcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 29; 2025: 27; 2026: 19.

85 papers.

Sort:

By area:

  1. 2025Reasoning Models Don't Always Say What They Thinkcited by 388
  2. 2023Explainability for Large Language Models: A Surveycited by 331
  3. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  4. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithfulcited by 166
  5. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  6. 2024Are self-explanations from Large Language Models faithful?cited by 101
  7. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  8. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  9. 2025Are DeepSeek R1 And Other Reasoning Models More Faithful?cited by 69
  10. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 57
  11. 2024A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?cited by 54
  12. 2024Designing a Dashboard for Transparency and Control of Conversational AIcited by 43
38 more
  1. 2025Measuring Chain of Thought Faithfulness by Unlearning Reasoning Stepscited by 43
  2. 2024Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thoughtcited by 37
  3. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  4. 2024Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuningcited by 26
  5. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  6. 2024Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs?cited by 24
  7. 2024Causality for Large Language Modelscited by 24
  8. 2025Reading Between the Prompts: How Stereotypes Shape LLM's Implicit Personalizationcited by 24
  9. 2024Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertaintycited by 23
  10. 2025Measuring the Faithfulness of Thinking Drafts in Large Reasoning Modelscited by 22
  11. 2023Towards Evaluating AI Systems for Moral Status Using Self-Reportscited by 21
  12. 2025Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoningcited by 19
  13. 2024FaithLM: Towards Faithful Explanations for Large Language Modelscited by 18
  14. 2024A Survey of AI-generated Text Forensic Systems: Detection, Attribution, and Characterizationcited by 18
  15. 2024Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learningcited by 18
  16. 2024Interpretable LLM-based Table Question Answeringcited by 18
  17. 2024Correctness is not Faithfulness in RAG Attributionscited by 18
  18. 2024xTower: A Multilingual LLM for Explaining and Correcting Translation Errorscited by 17
  19. 2025LLMs Don't Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual Explanationscited by 16
  20. 2026Counterfactual Simulation Training for Chain-of-Thought Faithfulnesscited by 15
  21. 2024GLIDER: Grading LLM Interactions and Decisions using Explainable Rankingcited by 14
  22. 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insightscited by 13
  23. 2023Self-Consistency of Large Language Models under Ambiguitycited by 12
  24. 2024Evaluating Human Alignment and Model Faithfulness of LLM Rationalecited by 12
  25. 2025A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Modelscited by 12
  26. 2026A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behaviorcited by 12
  27. 2023ALMANACS: A Simulatability Benchmark for Language Model Explainabilitycited by 11
  28. 2025Do Large Language Models Know How Much They Know?cited by 11
  29. 2025Robust Answers, Fragile Logic: Probing the Decoupling Hypothesis in LLM Reasoningcited by 11
  30. 2024Interpretability Needs a New Paradigmcited by 10
  31. 2025Can LLMs Explain Themselves Counterfactually?cited by 10
  32. 2024Why Would You Suggest That? Human Trust in Language Model Responsescited by 9
  33. 2025ConSim: Measuring Concept-Based Explanations' Effectiveness with Automated Simulatabilitycited by 9
  34. 2025LExT: Towards Evaluating Trustworthiness of Natural Language Explanationscited by 9
  35. 2023Controlled Generation with Prompt Insertion for Natural Language Explanations in Grammatical Error Correctioncited by 7
  36. 2025Advancing the Scientific Method with Large Language Models: From Hypothesis to Discoverycited by 7
  37. 2025Counterfactual Simulatability of LLM Explanations for Generation Taskscited by 7
  38. 2026Do LLM Self-Explanations Help Users Predict Model Behavior? Evaluating Counterfactual Simulatability with Pragmatic Perturbationscited by 7

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Eligibility of LLMs for Counterfactual Reasoning: A Decompositional Study
  2. 2024Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models
  3. 2024Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models
  4. 2024The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Models
  5. 2023CLOMO: Counterfactual Logical Modification with Large Language Models
  6. 2023Counterfactual reasoning: Testing language models' understanding of hypothetical scenarios
  7. 2025Flying Pigs, FaR and Beyond: Evaluating LLM Reasoning in Counterfactual Worlds
  8. 2026What LLMs explain is not what they believe: Evaluating explanation sufficiency under models' own input beliefs