# Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals

arXiv: https://arxiv.org/abs/2310.00603 · HTML: https://mlsanity.com/paper/2310.00603/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 94 references that are in the map.
- First posted on arXiv 1 Oct 2023.

## Builds on (most cited first, 30 of 60)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning](https://mlsanity.com/paper/1706.03461/) (2017)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Causal Reasoning and Large Language Models: Opening a New Frontier for Causality](https://mlsanity.com/paper/2305.00050/) (2023)
- [European Union regulations on algorithmic decision-making and a "right to explanation"](https://mlsanity.com/paper/1606.08813/) (2016)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)

## Built on by (most cited first, 30 of 44)

- [The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs](https://mlsanity.com/paper/2501.10970/) (2025)
- [On Measuring Faithfulness or Self-consistency of Natural Language Explanations](https://mlsanity.com/paper/2311.07466/) (2023)
- [Causal Inference with Large Language Model: A Survey](https://mlsanity.com/paper/2409.09822/) (2024)
- [Large Language Models and Causal Inference in Collaboration: A Survey](https://mlsanity.com/paper/2403.09606/) (2024)
- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [Causality for Large Language Models](https://mlsanity.com/paper/2410.15319/) (2024)
- [A Survey on Natural Language Counterfactual Generation](https://mlsanity.com/paper/2407.03993/) (2024)
- [Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance](https://mlsanity.com/paper/2410.18889/) (2024)
- [Can Slow-thinking LLMs Reason Over Time? Empirical Studies in Time Series Forecasting](https://mlsanity.com/paper/2505.24511/) (2025)
- [Large Language Models for Constrained-Based Causal Discovery](https://mlsanity.com/paper/2406.07378/) (2024)
- [Counterfactual Token Generation in Large Language Models](https://mlsanity.com/paper/2409.17027/) (2024)
- [Towards Faithful Natural Language Explanations: A Study Using Activation Patching in Large Language Models](https://mlsanity.com/paper/2410.14155/) (2024)
- [CausalEval: Towards Better Causal Reasoning in Language Models](https://mlsanity.com/paper/2410.16676/) (2024)
- [Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework](https://mlsanity.com/paper/2508.21422/) (2025)
- [Can LLMs Explain Themselves Counterfactually?](https://mlsanity.com/paper/2502.18156/) (2025)
- [CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstention](https://mlsanity.com/paper/2506.00519/) (2025)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [Interpreting Language Reward Models via Contrastive Explanations](https://mlsanity.com/paper/2411.16502/) (2024)
- [Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations](https://mlsanity.com/paper/2504.05294/) (2025)
- [Beyond Agreement: Diagnosing the Rationale Alignment of Automated Essay Scoring Methods based on Linguistically-informed Counterfactuals](https://mlsanity.com/paper/2405.19433/) (2024)
- [RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals](https://mlsanity.com/paper/2410.11348/) (2024)
- [Quantifying Trust: Financial Risk Management for Trustworthy AI Agents](https://mlsanity.com/paper/2604.03976/) (2026)
- [GraphNarrator: Generating Textual Explanations for Graph Neural Networks](https://mlsanity.com/paper/2410.15268/) (2024)
- [Can LLMs Reconcile Knowledge Conflicts in Counterfactual Reasoning](https://mlsanity.com/paper/2506.15732/) (2025)
- [Large Scale Foundation Models for Intelligent Manufacturing Applications: A Survey](https://mlsanity.com/paper/2312.06718/) (2023)
- [Using LLMs for Explaining Sets of Counterfactual Examples to Final Users](https://mlsanity.com/paper/2408.15133/) (2024)
- [NL-Eye: Abductive NLI for Images](https://mlsanity.com/paper/2410.02613/) (2024)
- [On the Eligibility of LLMs for Counterfactual Reasoning: A Decompositional Study](https://mlsanity.com/paper/2505.11839/) (2025)
- [Do explanations generalize across large reasoning models?](https://mlsanity.com/paper/2601.11517/) (2026)
- [Visual Inception: Compromising Long-term Planning in Agentic Recommenders via Multimodal Memory Poisoning](https://mlsanity.com/paper/2604.16966/) (2026)

## Abstract (arXiv)

Causal explanations of the predictions of NLP systems are essential to ensure safety and establish trust. Yet, existing methods often fall short of explaining model predictions effectively or efficiently and are often model-specific. In this paper, we address model-agnostic explanations, proposing two approaches for counterfactual (CF) approximation. The first approach is CF generation, where a large language model (LLM) is prompted to change a specific text concept while keeping confounding concepts unchanged. While this approach is demonstrated to be very effective, applying LLM at inference-time is costly. We hence present a second approach based on matching, and propose a method that is guided by an LLM at training-time and learns a dedicated embedding space. This space is faithful to a given causal graph and effectively serves to identify matches that approximate CFs. After showing theoretically that approximating CFs is required in order to construct faithful explanations, we benchmark our approaches and explain several models, including LLMs with billions of parameters. Our empirical results demonstrate the excellent performance of CF generation models as model-agnostic explainers. Moreover, our matching approach, which requires far less test-time resources, also provides effective explanations, surpassing many baselines. We also find that Top-K techniques universally improve every tested method. Finally, we showcase the potential of LLMs in constructing new benchmarks for model explanation and subsequently validate our conclusions. Our work illuminates new pathways for efficient and accurate approaches to interpreting NLP systems.