Data to 7 Oct 2026

Understanding Neural Networks through Representation Erasure

arXiv 1612.08220 ↗First posted 24 Dec 2016cs.CLcited by 223

Not on the map. Why?

Abstract (arXiv)

While neural networks have been successfully applied to many natural language processing tasks, they come at the cost of interpretability. In this paper, we propose a general methodology to analyze and interpret decisions from a neural model by observing the effects on the model of erasing various parts of the representation, such as input word-vector dimensions, intermediate hidden units, or input words. We present several approaches to analyzing the effects of such erasure, from computing the relative difference in evaluation metrics, to using reinforcement learning to erase the minimum set of input words in order to flip a neural model's decision. In a comprehensive analysis of multiple NLP tasks, including linguistic feature classification, sentence-level sentiment analysis, and document level sentiment aspect prediction, we show that the proposed methodology not only offers clear explanations about neural model decisions, but also provides a way to conduct error analysis on neural models.

Key facts. Cited by 223 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 64; 2022: 39; 2023: 37; 2024: 39; 2025: 23; 2026: 21.

223 papers.

Sort:

By area:

2023Explainability for Large Language Models: A Surveycited by 331
2021Benchmarking and Survey of Explanation Methods for Black Box Modelscited by 136
2021Scaling Up Influence Functionscited by 130
2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
2021BERT: A Review of Applications in Natural Language Processing and Understandingcited by 109
2022Interpreting Language Models with Contrastive Explanationscited by 91
2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 86
2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
2021ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanationcited by 78

38 more

2022Learning and Evaluating Graph Neural Network Explanations based on Counterfactual and Factual Reasoningcited by 78
2021Contrastive Explanations for Model Interpretabilitycited by 75
2021Triggerless Backdoor Attack for NLP Tasks with Clean Labelscited by 61
2022PINTO: Faithful Language Reasoning Using Prompt-Generated Rationalescited by 56
2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
2021What Context Features Can Transformer Language Models Use?cited by 49
2023FunnyBirds: A Synthetic Vision Dataset for a Part-Based Analysis of Explainable AI Methodscited by 49
2022Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvementcited by 48
2021Discretized Integrated Gradients for Explaining Language Modelscited by 45
2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
2022Rethinking Attention-Model Explainability through Faithfulness Violation Testcited by 43
2022How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysiscited by 42
2023Explaining How Transformers Use Context to Build Predictionscited by 42
2021Automatic Sleep Staging of EEG Signals: Recent Development, Challenges, and Future Directionscited by 41
2021Evaluating Saliency Methods for Neural Language Modelscited by 37
2022On the Explainability of Natural Language Processing Deep Modelscited by 37
2026SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agentscited by 37
2021Order in the Court: Explainable AI Methods Prone to Disagreementcited by 36
2021Understanding Interlocking Dynamics of Cooperative Rationalizationcited by 36
2021Fast Axiomatic Attribution for Neural Networkscited by 35
2023HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translationcited by 35
2021Civil Rephrases Of Toxic Texts With Self-Supervised Transformerscited by 32
2021UNIREX: A Unified Learning Framework for Language Model Rationale Extractioncited by 32
2022Global Concept-Based Interpretability for Graph Neural Networks via Neuron Analysiscited by 32
2025Interpreting the linear structure of vision-language model embedding spacescited by 32
2023Quantifying Uncertainty in Natural Language Explanations of Large Language Modelscited by 31
2021Attention Flows are Shapley Value Explanationscited by 30
2021Perturbing Inputs for Fragile Interpretations in Deep Natural Language Processingcited by 30
2021Explain and Predict, and then Predict Againcited by 29
2021Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classificationcited by 29
2023Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarkscited by 29
2022ViT-CX: Causal Explanation of Vision Transformerscited by 27
2021Rationales for Sequential Predictionscited by 26
2022ferret: a Framework for Benchmarking Explainers on Transformerscited by 26
2021Counterfactual Explanations for Models of Codecited by 24
2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24

50 most cited of 223 All 223 (JSON)