Not All Explanations Simulate Equally: Comparing Verbalized Feature Attributions and Self-Generated Rationales
On the map: Synthetic data and self-improvement for LLMs → Synthetic reasoning data and bootstrapping
Abstract (arXiv)
Natural-language explanations are often treated as a unified interface for understanding model behavior, but different explanation sources may support simulation in different ways. This paper compares two families of explanations for question answering models: verbalized feature attributions and self-generated rationales. We evaluate them under a shared counterfactual simulation setting, using an LLM judge as predictor and measuring whether it can better predict a model's answers to follow-up questions when given its explanation. Across multiple instruction-tuned models, we analyze how explanation source, verbalization strategy, and feature granularity affect the simulatability of explanations. Our results show that explanation format and granularity affect simulatability: attribution-based explanations and self-generated rationales differ in how much they improve counterfactual prediction, with effects that vary across models and formats.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Synthetic reasoning data and bootstrapping, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 38 of its 48 references that are in the map. First posted on arXiv 31 May 2026.
Builds on
38 of 48 references.
References · 38
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
- 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
- 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
- 2025OpenAI GPT-5 System Cardcited by 910
- 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
- 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
- 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
26 more
- 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
- 2016Rationalizing Neural Predictionscited by 281
- 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
- 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 189
- 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
- 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
- 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
- 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
- 2023Faithfulness Tests for Natural Language Explanationscited by 107
- 2020Explaining NLP Models via Minimal Contrastive Editing (MiCE)cited by 105
- 2024Are self-explanations from Large Language Models faithful?cited by 101
- 2020Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?cited by 95
- 2020Evaluating Explanations: How much do explanations from the teacher aid students?cited by 94
- 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
- 2023Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanationscited by 85
- 2018Do Explanations make VQA Models more Predictable to a Human?cited by 51
- 2025FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoningcited by 25
- 2023TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long Documentscited by 17
- 2022Saliency Map Verbalization: Comparing Feature Importance Representations from Model-free and Instruction-based Methodscited by 13
- 2026A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behaviorcited by 12
- 2025ConSim: Measuring Concept-Based Explanations' Effectiveness with Automated Simulatabilitycited by 9
- 2025Counterfactual Simulatability of LLM Explanations for Generation Taskscited by 7
- 2026Do LLM Self-Explanations Help Users Predict Model Behavior? Evaluating Counterfactual Simulatability with Pragmatic Perturbationscited by 7
- 2024Local Explanations and Self-Explanations for Assessing Faithfulness in black-box LLMscited by 2
- 2025ICX360: In-Context eXplainability 360 Toolkitcited by 2
- 2024A Systematic Comparison between Extractive Self-Explanations and Human Rationales in Text Classificationcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026From XAI to Stories: A Factorial Study of LLM-Generated Explanation Quality
- 2024Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models
- 2025LLMs Don't Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual Explanations
- 2026What LLMs explain is not what they believe: Evaluating explanation sufficiency under models' own input beliefs
- 2023What if you said that differently?: How Explanation Formats Affect Human Feedback Efficacy and User Perception
- 2026When Do Explanations Help In-Context Learning? A Comparative Study of Natural Language Explanation Types and Faithfulness
- 2024Evaluating Human Alignment and Model Faithfulness of LLM Rationale
- 2024The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Models