mlsanityData to 14 Oct 2026

Fine-grained Interpretation and Causation Analysis in Deep NLP Models

arXiv 2105.08039 ↗First posted 17 May 2021cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper is a write-up for the tutorial on "Fine-grained Interpretation and Causation Analysis in Deep NLP Models" that we are presenting at NAACL 2021. We present and discuss the research work on interpreting fine-grained components of a model from two perspectives, i) fine-grained interpretation, ii) causation analysis. The former introduces methods to analyze individual neurons and a group of neurons with respect to a language property or a task. The latter studies the role of neurons and input features in explaining decisions made by the model. We also discuss application of neuron analysis such as network manipulation and domain adaptation. Moreover, we present two toolkits namely NeuroX and Captum, that support functionalities discussed in this tutorial.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 47 of its 59 references that are in the map. First posted on arXiv 17 May 2021.

Builds on

47 of 59 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  5. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  6. 2018Deep contextualized word representationscited by 2,290
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  8. 2016The Mythos of Model Interpretabilitycited by 721
  9. 2019Designing and Interpreting Probes with Control Taskscited by 640
  10. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547
  11. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  12. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
35 more
  1. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  2. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 255
  3. 2018Targeted Syntactic Evaluation of Language Modelscited by 242
  4. 2016Understanding Neural Networks through Representation Erasurecited by 224
  5. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  6. 2018Colorless green recurrent networks dream hierarchicallycited by 213
  7. 2020A Survey of the State of Explainable AI for Natural Language Processingcited by 206
  8. 2015Visualizing and Understanding Neural Models in NLPcited by 200
  9. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 191
  10. 2020Compositional Explanations of Neuronscited by 172
  11. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  12. 2015Visualizing and Understanding Recurrent Networkscited by 160
  13. 2019The Shapley Taylor Interaction Indexcited by 143
  14. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  15. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 111
  16. 2020Explaining Explanations: Axiomatic Feature Interactions for Deep Networkscited by 103
  17. 2020Analyzing Individual Neurons in Pre-trained Language Modelscited by 97
  18. 2019The emergence of number and syntax units in LSTM language modelscited by 94
  19. 2018How Important Is a Neuron?cited by 73
  20. 2020How does this interaction affect me? Interpretable attribution for feature interactionscited by 73
  21. 2020Analyzing the Source and Target Contributions to Predictions in Neural Machine Translationcited by 71
  22. 2018Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Taskscited by 56
  23. 2020Similarity Analysis of Contextual Word Representation Modelscited by 54
  24. 2020Intrinsic Probing through Dimension Selectioncited by 45
  25. 2018The Importance of Being Recurrent for Modeling Hierarchical Structurecited by 42
  26. 2016Representation of linguistic form and function in recurrent neural networkscited by 39
  27. 2019On the Linguistic Representational Power of Neural Machine Translation Modelscited by 34
  28. 2020Asking without Telling: Exploring Latent Ontologies in Contextual Representationscited by 34
  29. 2018NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networkscited by 31
  30. 2020Finding Experts in Transformer Modelscited by 28
  31. 2019Discovery of Natural Language Concepts in Individual Units of CNNscited by 21
  32. 2018Interpretable Textual Neuron Representations for NLPcited by 16
  33. 2021Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translationcited by 15
  34. 2019Unsupervised Transfer Learning via BERT Neuron Selectioncited by 4
  35. 2018Ablation of a Robot's Brain: Neural Networks Under a Knifecited by 2

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 1.

4 papers.

  1. 2022Measuring the Mixing of Contextual Information in the Transformercited by 65
  2. 2022Analyzing Encoded Concepts in Transformer Language Modelscited by 38
  3. 2022A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inferencecited by 5
  4. 2023Computational modeling of semantic changecited by 2