# Evaluating Attribution Methods using White-Box LSTMs

arXiv: https://arxiv.org/abs/2010.08606 · HTML: https://mlsanity.com/paper/2010.08606/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2020.

## Built on by (most cited first, 8 of 8)

- ["Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification](https://mlsanity.com/paper/2111.07367/) (2021)
- [Evaluating Saliency Methods for Neural Language Models](https://mlsanity.com/paper/2104.05824/) (2021)
- [DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformers](https://mlsanity.com/paper/2310.03686/) (2023)
- [On Sample Based Explanation Methods for NLP:Efficiency, Faithfulness, and Semantic Evaluation](https://mlsanity.com/paper/2106.04753/) (2021)
- [A Close Look at Decomposition-based XAI-Methods for Transformer Language Models](https://mlsanity.com/paper/2502.15886/) (2025)
- [Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distribution](https://mlsanity.com/paper/2310.14840/) (2023)
- [Feature Interactions Reveal Linguistic Structure in Language Models](https://mlsanity.com/paper/2306.12181/) (2023)
- [What Exactly do Children Receive in Language Acquisition? A Case Study on CHILDES with Automated Detection of Filler-Gap Dependencies](https://mlsanity.com/paper/2603.02082/) (2026)

## Abstract (arXiv)

Interpretability methods for neural networks are difficult to evaluate because we do not understand the black-box models typically used to test them. This paper proposes a framework in which interpretability methods are evaluated using manually constructed networks, which we call white-box networks, whose behavior is understood a priori. We evaluate five methods for producing attribution heatmaps by applying them to white-box LSTM classifiers for tasks based on formal languages. Although our white-box classifiers solve their tasks perfectly and transparently, we find that all five attribution methods fail to produce the expected model explanations.