mlsanityData to 14 Oct 2026

Sequential Integrated Gradients: a simple but effective method for explaining language models

arXiv 2305.15853 ↗First posted 25 May 2023cs.CL cs.AI cs.LGcited by 52

Not on the map. Why?

Abstract (arXiv)

Several explanation methods such as Integrated Gradients (IG) can be characterised as path-based methods, as they rely on a straight line between the data and an uninformative baseline. However, when applied to language models, these methods produce a path for each word of a sentence simultaneously, which could lead to creating sentences from interpolated words either having no clear meaning, or having a significantly different meaning compared to the original sentence. In order to keep the meaning of these sentences as close as possible to the original one, we propose Sequential Integrated Gradients (SIG), which computes the importance of each word in a sentence by keeping fixed every other words, only creating interpolations between the baseline and the word of interest. Moreover, inspired by the training procedure of several language models, we also propose to replace the baseline token "pad" with the trained token "mask". While being a simple improvement over the original IG method, we show on various models and datasets that SIG proves to be a very effective method for explaining language models.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 18 of its 31 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

18 of 31 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  9. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547
  10. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
  11. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 297
  12. 2019Explanations can be manipulated and geometry is to blamecited by 216
6 more
  1. 2019BERT for Joint Intent Classification and Slot Fillingcited by 196
  2. 2018Explaining Explanations in AIcited by 137
  3. 2019Fine-tune BERT for Extractive Summarizationcited by 122
  4. 2021Counterfactual Explanations Can Be Manipulatedcited by 113
  5. 2022A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributionscited by 64
  6. 2021Discretized Integrated Gradients for Explaining Language Modelscited by 46

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 18; 2025: 19; 2026: 9.

52 papers.

Sort:

By area:

  1. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  2. 2023Explainability for Large Language Models: A Surveycited by 333
  3. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  4. 2024ContextCite: Attributing Model Generation to Contextcited by 98
  5. 2023Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neuronscited by 56
  6. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  7. 2024From Understanding to Utilization: A Survey on Explainability for Large Language Modelscited by 48
  8. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 46
  9. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  10. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 25
  11. 2024AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attributioncited by 23
  12. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
38 more
  1. 2024LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools and Self-Explanationscited by 16
  2. 2025Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associationscited by 15
  3. 2024Knowledge Localization: Mission Not Accomplished? Enter Query Localization!cited by 13
  4. 2025Learning to Attribute with Attentioncited by 13
  5. 2024Adversarial Representation Engineering: A General Model Editing Framework for Large Language Modelscited by 12
  6. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  7. 2025Reasoning-Grounded Natural Language Explanations for Language Modelscited by 5
  8. 2023Time Interpret: a Unified Model Interpretability Library for Time Seriescited by 4
  9. 2025B-cos LM: Efficiently Transforming Pre-trained Language Models for Improved Explainabilitycited by 4
  10. 2025Explanations of Large Language Models Explain Language Representations in the Braincited by 4
  11. 2025Who Owns the Output? Bridging Law and Technology in LLMs Attributioncited by 4
  12. 2025Noiser: Bounded Input Perturbations for Attributing Large Language Modelscited by 4
  13. 2025From Directions to Cones: Exploring Multidimensional Representations of Propositional Facts in LLMscited by 4
  14. 2023A New Baseline Assumption of Integated Gradients Based on Shaply valuecited by 3
  15. 2024Counterfactuals As a Means for Evaluating Faithfulness of Attribution Methods in Autoregressive Language Modelscited by 3
  16. 2024One Mind, Many Tongues: A Deep Dive into Language-Agnostic Knowledge Neurons in Large Language Modelscited by 3
  17. 2025Explaining the Reasoning of Large Language Models Using Attribution Graphscited by 3
  18. 2024"Sorry, Come Again?" Prompting -- Enhancing Comprehension and Diminishing Hallucination with [PAUSE]-injected Optimal Paraphrasingcited by 2
  19. 2025Mitigating Multimodal Hallucinations via Gradient-based Self-Reflectioncited by 2
  20. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanationscited by 2
  21. 2026From Heuristic Selection to Automated Algorithm Design: LLMs Benefit from Strong Priorscited by 2
  22. 2026Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Rewardcited by 2
  23. 2025Towards Structurally Explainable Machine-Generated Text Detection: A Graph-Perspective Frameworkcited by 1
  24. 2025Failure Modes of Time Series Interpretability Algorithms for Critical Care Applications and Potential Solutionscited by 1
  25. 2025Path-Weighted Integrated Gradients for Interpretable Dementia Classificationcited by 1
  26. 2026DRIV-EX: Counterfactual Explanations for Driving LLMscited by 1
  27. 2024PE: A Poincare Explanation Method for Fast Text Hierarchy Generationcited by 0
  28. 2024Unveiling LLM Mechanisms Through Neural ODEs and Control Theorycited by 0
  29. 2024Unraveling Text Generation in LLMs: A Stochastic Differential Equation Approachcited by 0
  30. 2024Advancing Fairness in Natural Language Processing: From Traditional Methods to Explainabilitycited by 0
  31. 2024GUIDEQ: Framework for Guided Questioning for progressive informational collection and classificationcited by 0
  32. 2025Application of integrated gradients explainability to sociopsychological semantic markerscited by 0
  33. 2025Attribution assignment for deep-generative sequence models enables interpretability analysis using positive-only datacited by 0
  34. 2025Explaining Fine Tuned LLMs via Counterfactuals A Knowledge Graph Driven Frameworkcited by 0
  35. 2025Bridging Fairness and Explainability: Can Input-Based Explanations Promote Fairness in Hate Speech Detection?cited by 0
  36. 2026Select or Project? Evaluating Lower-dimensional Vectors for LLM Training Data Explanationscited by 0
  37. 2026The Attribution Contract for Generative Language Modelscited by 0
  38. 2026Explaining Black-Box Language Models: Learning to Optimize Linguistically-Structured Word Subsetscited by 0

50 most cited of 52 All 52 (JSON)