# Double Trouble: How to not explain a text classifier's decisions using counterfactuals synthesized by masked language models?

arXiv: https://arxiv.org/abs/2110.11929 · HTML: https://mlsanity.com/paper/2110.11929/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 44 references that are in the map.
- First posted on arXiv 22 Oct 2021.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [Real Time Image Saliency for Black Box Classifiers](https://mlsanity.com/paper/1705.07857/) (2017)
- [Understanding Neural Networks through Representation Erasure](https://mlsanity.com/paper/1612.08220/) (2016)
- [On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection](https://mlsanity.com/paper/1811.07901/) (2018)
- [Explaining Image Classifiers by Counterfactual Generation](https://mlsanity.com/paper/1807.08024/) (2018)
- [Pathologies of Neural Models Make Interpretations Difficult](https://mlsanity.com/paper/1804.07781/) (2018)
- [Explaining Classifiers with Causal Concept Effect (CaCE)](https://mlsanity.com/paper/1907.07165/) (2019)
- [Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing](https://mlsanity.com/paper/2102.12060/) (2021)
- [Explaining Recurrent Neural Network Predictions in Sentiment Analysis](https://mlsanity.com/paper/1706.07206/) (2017)
- [The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations](https://mlsanity.com/paper/2106.00786/) (2021)
- [Neural Network Attributions: A Causal Perspective](https://mlsanity.com/paper/1902.02302/) (2019)
- [Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?](https://mlsanity.com/paper/2012.15180/) (2020)
- [Incorporating Priors with Feature Attribution on Text Classification](https://mlsanity.com/paper/1906.08286/) (2019)
- [Quantifying Interpretability and Trust in Machine Learning Systems](https://mlsanity.com/paper/1901.08558/) (2019)
- [Interpretation of NLP models through input marginalization](https://mlsanity.com/paper/2010.13984/) (2020)
- [Feature Removal Is a Unifying Principle for Model Explanation Methods](https://mlsanity.com/paper/2011.03623/) (2020)
- [Considering Likelihood in NLP Classification Explanations with Occlusion and Language Modeling](https://mlsanity.com/paper/2004.09890/) (2020)

## Built on by (most cited first, 5 of 5)

- [Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations](https://mlsanity.com/paper/2310.11207/) (2023)
- [Necessity and Sufficiency for Explaining Text Classifiers: A Case Study in Hate Speech Detection](https://mlsanity.com/paper/2205.03302/) (2022)
- [The Solvability of Interpretability Evaluation Metrics](https://mlsanity.com/paper/2205.08696/) (2022)
- [Multi-Level Explanations for Generative Language Models](https://mlsanity.com/paper/2403.14459/) (2024)
- [How explainable are adversarially-robust CNNs?](https://mlsanity.com/paper/2205.13042/) (2022)

## Abstract (arXiv)

A principle behind dozens of attribution methods is to take the prediction difference between before-and-after an input feature (here, a token) is removed as its attribution. A popular Input Marginalization (IM) method (Kim et al., 2020) uses BERT to replace a token, yielding more plausible counterfactuals. While Kim et al. (2020) reported that IM is effective, we find this conclusion not convincing as the DeletionBERT metric used in their paper is biased towards IM. Importantly, this bias exists in Deletion-based metrics, including Insertion, Sufficiency, and Comprehensiveness. Furthermore, our rigorous evaluation using 6 metrics and 3 datasets finds no evidence that IM is better than a Leave-One-Out (LOO) baseline. We find two reasons why IM is not better than LOO: (1) deleting a single word from the input only marginally reduces a classifier's accuracy; and (2) a highly predictable word is always given near-zero attribution, regardless of its true importance to the classifier. In contrast, making LIME samples more natural via BERT consistently improves LIME accuracy under several ROAR metrics.