mlsanityData to 14 Oct 2026

Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs

arXiv 1801.05453 ↗First posted 16 Jan 2018cs.CL cs.LG stat.MLcited by 67

Not on the map. Why?

Abstract (arXiv)

The driving force behind the recent success of LSTMs has been their ability to learn complex and non-linear relationships. Consequently, our inability to describe these relationships has led to LSTMs being characterized as black boxes. To this end, we introduce contextual decomposition (CD), an interpretation algorithm for analysing individual predictions made by standard LSTMs, without any changes to the underlying model. By decomposing the output of a LSTM, CD captures the contributions of combinations of words or variables to the final prediction of an LSTM. On the task of sentiment analysis with the Yelp and SST data sets, we show that CD is able to reliably identify words and phrases of contrasting sentiment, and how they are combined to yield the LSTM's final prediction. Using the phrase-level labels in SST, we also demonstrate that CD is able to successfully extract positive and negative negations from an LSTM, something which has not previously been done.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Jan 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 23; 2022: 14; 2023: 18; 2024: 4; 2025: 6; 2026: 2.

67 papers.

Sort:

By area:

  1. 2022From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagationcited by 141
  2. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  3. 2021Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Casescited by 77
  4. 2023SHAP-IQ: Unified Approximation of any-order Shapley Interactionscited by 51
  5. 2022Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvementcited by 48
  6. 2021Discretized Integrated Gradients for Explaining Language Modelscited by 46
  7. 2021Shapley Explanation Networkscited by 41
  8. 2021Defining and Quantifying the Emergence of Sparse Concepts in DNNscited by 36
  9. 2021Fast Axiomatic Attribution for Neural Networkscited by 35
  10. 2023Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Modelscited by 35
  11. 2021Adaptive wavelet distillation from neural networks through interpretationscited by 32
  12. 2022FaiRR: Faithful and Robust Deductive Reasoning over Natural Languagecited by 29
38 more
  1. 2023Does a Neural Network Really Encode Symbolic Concepts?cited by 27
  2. 2024Efficient Automated Circuit Discovery in Transformers using Contextual Decompositioncited by 23
  3. 2024KernelSHAP-IQ: Weighted Least-Square Optimization for Shapley Interactionscited by 22
  4. 2021Interpreting Deep Learning Models in Natural Language Processing: A Reviewcited by 19
  5. 2021Logic Traps in Evaluating Attribution Scorescited by 18
  6. 2023DEGREE: Decomposition Based Explanation For Graph Neural Networkscited by 18
  7. 2023Uncovering hidden geometry in Transformers via disentangling position and contextcited by 18
  8. 2021A Unified Game-Theoretic Interpretation of Adversarial Robustnesscited by 17
  9. 2023Reveal to Revise: An Explainable AI Life Cycle for Iterative Bias Correction of Deep Modelscited by 17
  10. 2021Explainable Deep Learning in Healthcare: A Methodological Survey from an Attribution Viewcited by 15
  11. 2022A Fine-grained Interpretability Evaluation Benchmark for Neural NLPcited by 15
  12. 2023Towards Fixing Clever-Hans Predictors with Counterfactual Knowledge Distillationcited by 14
  13. 2021Interpreting Attributions and Interactions of Adversarial Attackscited by 13
  14. 2022Self-explaining deep models with logic rule reasoningcited by 13
  15. 2021Explaining Neural Network Predictions on Sentence Pairs via Learning Word-Group Maskscited by 12
  16. 2024Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanationscited by 11
  17. 2025Explainability in Practice: A Survey of Explainable NLP Across Various Domainscited by 11
  18. 2021Can We Faithfully Represent Masked States to Compute Shapley Values on a DNN?cited by 10
  19. 2023Learning with Explanation Constraintscited by 8
  20. 2023Feature Interactions Reveal Linguistic Structure in Language Modelscited by 8
  21. 2023Explaining with Contrastive Phrasal Highlighting: A Case Study in Assisting Humans to Detect Translation Differencescited by 8
  22. 2021Local Explanation of Dialogue Response Generationcited by 7
  23. 2021Machine Learning for Multimodal Electronic Health Records-based Research: Challenges and Perspectivescited by 7
  24. 2023On the Interplay between Fairness and Explainabilitycited by 7
  25. 2023Proto-lm: A Prototypical Network-Based Framework for Built-in Interpretability in Large Language Modelscited by 7
  26. 2021Towards Learning an Unbiased Classifier from Biased Data via Conditional Adversarial Debiasingcited by 6
  27. 2023Improving Interpretability via Explicit Word Interaction Graph Layercited by 6
  28. 2021Interpretable Artificial Intelligence through the Lens of Feature Interactioncited by 5
  29. 2021Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifierscited by 5
  30. 2022ReX: A Framework for Incorporating Temporal Information in Model-Agnostic Local Explanation Techniquescited by 5
  31. 2022Power of Explanations: Towards automatic debiasing in hate speech detectioncited by 5
  32. 2022Feature Importance for Time Series Data: Improving KernelSHAPcited by 5
  33. 2022Implicit N-grams Induced by Recurrencecited by 4
  34. 2021On the Post-hoc Explainability of Deep Echo State Networks for Time Series Forecasting, Image and Video Classificationcited by 3
  35. 2021Attention vs non-attention for a Shapley-based explanation methodcited by 3
  36. 2023Improving Prototypical Visual Explanations with Reward Reweighing, Reselection, and Retrainingcited by 3
  37. 2022Right for the Right Latent Factors: Debiasing Generative Models via Disentanglementcited by 2
  38. 2023A semantically enhanced dual encoder for aspect sentiment triplet extractioncited by 2

50 most cited of 67 All 67 (JSON)