# AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models

arXiv: https://arxiv.org/abs/1909.09251 · HTML: https://mlsanity.com/paper/1909.09251/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2019.

## Built on by (most cited first, 30 of 74)

- [What the DAAM: Interpreting Stable Diffusion Using Cross Attention](https://mlsanity.com/paper/2210.04885/) (2022)
- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [MoEfication: Transformer Feed-forward Layers are Mixtures of Experts](https://mlsanity.com/paper/2110.01786/) (2021)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Robustness Gym: Unifying the NLP Evaluation Landscape](https://mlsanity.com/paper/2101.04840/) (2021)
- [Measure and Improve Robustness in NLP Models: A Survey](https://mlsanity.com/paper/2112.08313/) (2021)
- [XAI for Transformers: Better Explanations through Conservative Propagation](https://mlsanity.com/paper/2202.07304/) (2022)
- [Interpreting Language Models with Contrastive Explanations](https://mlsanity.com/paper/2202.10419/) (2022)
- [Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilities](https://mlsanity.com/paper/2308.12833/) (2023)
- [ExplainaBoard: An Explainable Leaderboard for NLP](https://mlsanity.com/paper/2104.06387/) (2021)
- [Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps](https://mlsanity.com/paper/2204.11073/) (2022)
- [First is Better Than Last for Language Data Influence](https://mlsanity.com/paper/2202.11844/) (2022)
- [Analyzing Chain-of-Thought Prompting in Large Language Models via Gradient-based Feature Attributions](https://mlsanity.com/paper/2307.13339/) (2023)
- [ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writing](https://mlsanity.com/paper/2305.09770/) (2023)
- [Exploring the Role of BERT Token Representations to Explain Sentence Probing Results](https://mlsanity.com/paper/2104.01477/) (2021)
- [LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models](https://mlsanity.com/paper/2204.12130/) (2022)
- [ferret: a Framework for Benchmarking Explainers on Transformers](https://mlsanity.com/paper/2208.01575/) (2022)
- [On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs](https://mlsanity.com/paper/2407.19200/) (2024)
- [Challenges in Applying Explainability Methods to Improve the Fairness of NLP Models](https://mlsanity.com/paper/2206.03945/) (2022)
- [Interpreting Deep Learning Models in Natural Language Processing: A Review](https://mlsanity.com/paper/2110.10470/) (2021)
- [Logic Traps in Evaluating Attribution Scores](https://mlsanity.com/paper/2109.05463/) (2021)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [On the Lack of Robust Interpretability of Neural Text Classifiers](https://mlsanity.com/paper/2106.04631/) (2021)
- [LMdiff: A Visual Diff Tool to Compare Language Models](https://mlsanity.com/paper/2111.01582/) (2021)
- [IMBERT: Making BERT Immune to Insertion-based Backdoor Attacks](https://mlsanity.com/paper/2305.16503/) (2023)
- [Saliency Map Verbalization: Comparing Feature Importance Representations from Model-free and Instruction-based Methods](https://mlsanity.com/paper/2210.07222/) (2022)
- [Token-Modification Adversarial Attacks for Natural Language Processing: A Survey](https://mlsanity.com/paper/2103.00676/) (2021)
- [T3-Vis: a visual analytic framework for Training and fine-Tuning Transformers in NLP](https://mlsanity.com/paper/2108.13587/) (2021)
- [The Solvability of Interpretability Evaluation Metrics](https://mlsanity.com/paper/2205.08696/) (2022)
- [Automatic Error Analysis for Document-level Information Extraction](https://mlsanity.com/paper/2209.07442/) (2022)

## Abstract (arXiv)

Neural NLP models are increasingly accurate but are imperfect and opaque---they break in counterintuitive ways and leave end users puzzled at their behavior. Model interpretation methods ameliorate this opacity by providing explanations for specific model predictions. Unfortunately, existing interpretation codebases make it difficult to apply these methods to new models and tasks, which hinders adoption for practitioners and burdens interpretability researchers. We introduce AllenNLP Interpret, a flexible framework for interpreting NLP models. The toolkit provides interpretation primitives (e.g., input gradients) for any AllenNLP model and task, a suite of built-in interpretation methods, and a library of front-end visualization components. We demonstrate the toolkit's flexibility and utility by implementing live demos for five interpretation methods (e.g., saliency maps and adversarial attacks) on a variety of models and tasks (e.g., masked language modeling using BERT and reading comprehension using BiDAF). These demos, alongside our code and tutorials, are available at https://allennlp.org/interpret .