mlsanityData to 7 Oct 2026

Visualizing and Understanding Neural Models in NLP

arXiv 1506.01066 ↗First posted 2 Jun 2015cs.CLcited by 199

Not on the map. Why?

Abstract (arXiv)

While neural networks have been successfully applied to many NLP tasks the resulting vector-based models are very difficult to interpret. For example it's not clear how they achieve em compositionality, building sentence meaning from the meanings of words and phrases. In this paper we describe four strategies for visualizing compositionality in neural models for NLP, inspired by similar work in computer vision. We first plot unit values to visualize compositionality of negation, intensification, and concessive clauses, allow us to see well-known markedness asymmetries in negation. We then introduce three simple and straightforward methods for visualizing a unit's em salience, the amount it contributes to the final composed meaning: (1) gradient back-propagation, (2) the variance of a token from the average word node, (3) LSTM-style gates that measure information flow. We test our methods on sentiment using simple recurrent nets and LSTMs. Our general-purpose methods may have wide applications for understanding compositionality and other semantic properties of deep networks , and also shed light on why LSTMs outperform simple recurrent nets,

Key facts. Cited by 199 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Jun 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 49; 2022: 41; 2023: 36; 2024: 31; 2025: 22; 2026: 20.

199 papers.

Sort:

By area:

  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  2. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  3. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  4. 2023Explainability for Large Language Models: A Surveycited by 331
  5. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155
  6. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  7. 2024Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Modelscited by 104
  8. 2021An Interpretability Illusion for BERTcited by 94
  9. 2024ContextCite: Attributing Model Generation to Contextcited by 93
  10. 2022Interpreting Language Models with Contrastive Explanationscited by 91
  11. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  12. 2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 86
38 more
  1. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  2. 2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
  3. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  4. 2021Triggerless Backdoor Attack for NLP Tasks with Clean Labelscited by 61
  5. 2023Multiscale Positive-Unlabeled Detection of AI-Generated Textscited by 59
  6. 2022PINTO: Faithful Language Reasoning Using Prompt-Generated Rationalescited by 56
  7. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  8. 2021On Guaranteed Optimal Robust Explanations for NLP Modelscited by 54
  9. 2024Identifying Semantic Induction Heads to Understand In-Context Learningcited by 46
  10. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  11. 2021SelfExplain: A Self-Explaining Architecture for Neural Text Classifierscited by 44
  12. 2022GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformerscited by 44
  13. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  14. 2023Explaining How Transformers Use Context to Build Predictionscited by 42
  15. 2022The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanationscited by 41
  16. 2021SSMix: Saliency-Based Span Mixup for Text Classificationcited by 39
  17. 2021Evaluating Saliency Methods for Neural Language Modelscited by 37
  18. 2022Analyzing Encoded Concepts in Transformer Language Modelscited by 37
  19. 2022On the Explainability of Natural Language Processing Deep Modelscited by 37
  20. 2021Understanding Interlocking Dynamics of Cooperative Rationalizationcited by 36
  21. 2022Missingness Bias in Model Debuggingcited by 36
  22. 2024Trust in AI: Progress, Challenges, and Future Directionscited by 35
  23. 2021A Study of Automatic Metrics for the Evaluation of Natural Language Explanationscited by 34
  24. 2022ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Datacited by 34
  25. 2023How are Prompts Different in Terms of Sensitivity?cited by 34
  26. 2024Gradient based Feature Attribution in Explainable AI: A Technical Reviewcited by 34
  27. 2021Explainable AI: current status and future directionscited by 32
  28. 2021UNIREX: A Unified Learning Framework for Language Model Rationale Extractioncited by 32
  29. 2021Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retrainingcited by 31
  30. 2022On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliencycited by 31
  31. 2023DecompX: Explaining Transformers Decisions by Propagating Token Decompositioncited by 31
  32. 2023Quantifying Uncertainty in Natural Language Explanations of Large Language Modelscited by 31
  33. 2023DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructionscited by 31
  34. 2021On the Sensitivity and Stability of Model Interpretations in NLPcited by 30
  35. 2021Perturbing Inputs for Fragile Interpretations in Deep Natural Language Processingcited by 30
  36. 2023Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammarscited by 30
  37. 2021DeepHate: Hate Speech Detection via Multi-Faceted Text Representationscited by 29
  38. 2021Exploring the Role of BERT Token Representations to Explain Sentence Probing Resultscited by 29

50 most cited of 199 All 199 (JSON)