mlsanityData to 13 Oct 2026

Counterfactual Simulatability of LLM Explanations for Generation Tasks

arXiv 2505.21740 ↗First posted 27 May 2025cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

LLMs can be unpredictable, as even slight alterations to the prompt can cause the output to change in unexpected ways. Thus, the ability of models to accurately explain their behavior is critical, especially in high-stakes settings. One approach for evaluating explanations is counterfactual simulatability, how well an explanation allows users to infer the model's output on related counterfactuals. Counterfactual simulatability has been previously studied for yes/no question answering tasks. We provide a general framework for extending this method to generation tasks, using news summarization and medical suggestion as example use cases. We find that while LLM explanations do enable users to better predict LLM outputs on counterfactuals in the summarization setting, there is significant room for improvement for medical suggestion. Furthermore, our results suggest that the evaluation for counterfactual simulatability may be more appropriate for skill-based tasks as opposed to knowledge-based tasks.

Key facts. Cited by 8 papers in the map (data through 13 Oct 2026). Builds on 23 of its 42 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

23 of 42 references.

References · 23

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,068
  3. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,101
  4. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 810
  5. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 526
  6. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  7. 2023Faithful Chain-of-Thought Reasoningcited by 286
  8. 2021Societal Biases in Language Generation: Progress and Challengescited by 202
  9. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 119
  10. 2024What Large Language Models Know and What People Think They Knowcited by 110
  11. 2024Are self-explanations from Large Language Models faithful?cited by 101
  12. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
11 more
  1. 2023Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanationscited by 85
  2. 2022Generating Scientific Claims for Zero-Shot Scientific Fact Checkingcited by 59
  3. 2023Post Hoc Explanations of Language Models Can Improve Language Modelscited by 50
  4. 2024Properties and Challenges of LLM-Generated Explanationscited by 24
  5. 2022WikiWhy: Answering and Explaining Cause-and-Effect Questionscited by 21
  6. 2021Probing Commonsense Explanation in Dialogue Response Generationcited by 13
  7. 2024Explainable AI improves task performance in human-AI collaborationcited by 12
  8. 2024Evaluating Human Alignment and Model Faithfulness of LLM Rationalecited by 12
  9. 2024Faithful and Plausible Natural Language Explanations for Image Classification: A Pipeline Approachcited by 7
  10. 2023Self-Explanation Prompting Improves Dialogue Understanding in Large Language Modelscited by 5
  11. 2024Local Explanations and Self-Explanations for Assessing Faithfulness in black-box LLMscited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Counterfactual Simulation Training for Chain-of-Thought Faithfulnesscited by 17
  2. 2026A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behaviorcited by 13
  3. 2026Do LLM Self-Explanations Help Users Predict Model Behavior? Evaluating Counterfactual Simulatability with Pragmatic Perturbationscited by 7
  4. 2026Not All Explanations Simulate Equally: Comparing Verbalized Feature Attributions and Self-Generated Rationalescited by 1
  5. 2026Justified or Just Convincing? Error Verifiability as a Dimension of LLM Qualitycited by 0
  6. 2026Limitations of Automated Simulatability: LLM Simulators Can Bypass Explanationscited by 0
  7. 2026Empath: Tracing Multi-Level Emotion Dynamics in Crisis Counseling Dialoguescited by 0
  8. 2026Efficient Reasoning Training Does Not Always Harm CoT Faithfulness and Monitorabilitycited by 0