mlsanityData to 7 Oct 2026

Measuring the Mixing of Contextual Information in the Transformer

arXiv 2203.04212 ↗First posted 8 Mar 2022cs.CLcited by 63

Not on the map. Why?

Abstract (arXiv)

The Transformer architecture aggregates input information through the self-attention mechanism, but there is no clear understanding of how this information is mixed across the entire model. Additionally, recent works have demonstrated that attention weights alone are not enough to describe the flow of information. In this paper, we consider the whole attention block -- multi-head attention, residual connection, and layer normalization -- and define a metric to measure token-to-token interactions within each layer. Then, we aggregate layer-wise interpretations to provide input attribution scores for model predictions. Experimentally, we show that our method, ALTI (Aggregation of Layer-wise Token-to-token Interactions), provides more faithful explanations and increased robustness than gradient-based methods.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 44 of its 58 references that are in the map. First posted on arXiv 8 Mar 2022.

Builds on

44 of 58 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  9. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  12. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
32 more
  1. 2019Attention is not Explanationcited by 916
  2. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  3. 2020Quantifying Attention Flow in Transformerscited by 759
  4. 2019Attention is not not Explanationcited by 629
  5. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  6. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
  7. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  8. 2019Is Attention Interpretable?cited by 387
  9. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  10. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  11. 2019Revealing the Dark Secrets of BERTcited by 300
  12. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 295
  13. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  14. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  15. 2019Assessing BERT's Syntactic Abilitiescited by 191
  16. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 158
  17. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155
  18. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  19. 2019On Identifiability in Transformerscited by 115
  20. 2019Learning to Deceive with Attention-Based Explanationscited by 111
  21. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  22. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  23. 2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66
  24. 2014Extraction of Salient Sentences from Labelled Documentscited by 50
  25. 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 50
  26. 2020Positional Artefacts Propagate Through Masked Language Model Embeddingscited by 34
  27. 2021Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retrainingcited by 31
  28. 2021On the Lack of Robust Interpretability of Neural Text Classifierscited by 17
  29. 2020Telling BERT's full story: from Local Attention to Global Aggregationcited by 15
  30. 2020Influence Patterns for Explaining Information Flow in BERTcited by 14
  31. 2021Effective Attention Sheds Light On Interpretabilitycited by 11
  32. 2021Fine-grained Interpretation and Causation Analysis in Deep NLP Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 11; 2024: 19; 2025: 20; 2026: 9.

63 papers.

Sort:

By area:

  1. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 92
  2. 2024From Understanding to Utilization: A Survey on Explainability for Large Language Modelscited by 47
  3. 2022Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformercited by 45
  4. 2023Quantifying Context Mixing in Transformerscited by 43
  5. 2023Explaining How Transformers Use Context to Build Predictionscited by 42
  6. 2023Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Mapscited by 33
  7. 2023DecompX: Explaining Transformers Decisions by Propagating Token Decompositioncited by 31
  8. 2024InternalInspector I2: Robust Confidence Estimation in LLMs through Internal Statescited by 29
  9. 2022Toxicity in Multilingual Machine Translation at Scalecited by 26
  10. 2022ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Modelscited by 25
  11. 2024LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Modelscited by 16
  12. 2023Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformerscited by 13
38 more
  1. 2024Evaluating Human Alignment and Model Faithfulness of LLM Rationalecited by 12
  2. 2025The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentationcited by 11
  3. 2025A Close Look at Decomposition-based XAI-Methods for Transformer Language Modelscited by 10
  4. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  5. 2023Feature Interactions Reveal Linguistic Structure in Language Modelscited by 8
  6. 2024Better Explain Transformers by Illuminating Important Informationcited by 8
  7. 2023Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attentioncited by 7
  8. 2025Are We Paying Attention to Her? Investigating Gender Disambiguation and Attention in Machine Translationcited by 7
  9. 2023Quantifying the Plausibility of Context Reliance in Neural Machine Translationcited by 6
  10. 2024Unveiling and Manipulating Prompt Influence in Large Language Modelscited by 6
  11. 2025Mitigating Hallucinated Translations in Large Language Models with Hallucination-focused Preference Optimizationcited by 6
  12. 2026Towards Long-Horizon Interpretability: Efficient and Faithful Multi-Token Attribution for Reasoning LLMscited by 6
  13. 2026Instruction Anchor: Dissecting the Mechanistic Dynamics of Modality Arbitrationcited by 6
  14. 2022ReX: A Framework for Incorporating Temporal Information in Model-Agnostic Local Explanation Techniquescited by 5
  15. 2024Context-Aware or Context-Insensitive? Assessing LLMs' Performance in Document-Level Translationcited by 5
  16. 2024LibraGrad: Balancing Gradient Flow for Universally Better Vision Transformer Attributionscited by 5
  17. 2026Geometry-Calibrated Conformal Abstention for Language Modelscited by 5
  18. 2024Isotropy, Clusters, and Classifierscited by 4
  19. 2024NormXLogit: The Head-on-Top Never Liescited by 4
  20. 2025Understanding LLMs' Cross-Lingual Context Retrieval: How Good It Is And Where It Comes Fromcited by 4
  21. 2025Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreementcited by 4
  22. 2024Counterfactuals As a Means for Evaluating Faithfulness of Attribution Methods in Autoregressive Language Modelscited by 3
  23. 2025CueTip: An Interactive and Explainable Physics-aware Pool Assistantcited by 3
  24. 2025LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Modelscited by 3
  25. 2023Computational modeling of semantic changecited by 2
  26. 2023Why bother with geometry? On the relevance of linear decompositions of Transformer embeddingscited by 2
  27. 2025Which Words Matter Most in Zero-Shot Prompts?cited by 2
  28. 2025Steering Information Utility in Key-Value Memory for Language Model Post-Trainingcited by 2
  29. 2026GRAIL: Gradient-Reweighted Advantages for Reinforcement Learning with Verifiable Rewardscited by 2
  30. 2024Explanation Regularisation through the Lens of Attributionscited by 1
  31. 2024Unveiling the Role of Pretraining in Direct Speech Translationcited by 1
  32. 2024How Language Models Prioritize Contextual Grammatical Cues?cited by 1
  33. 2024Analyzing Context Contributions in LLM-based Machine Translationcited by 1
  34. 2024On Explaining with Attention Matricescited by 1
  35. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1
  36. 2025DePass: Unified Feature Attributing by Simple Decomposed Forward Passcited by 1
  37. 2025Order-Level Attention Similarity Across Language Models: A Latent Commonalitycited by 1
  38. 2026Layer-wise Positional Bias in Short-Context Language Modelingcited by 1

50 most cited of 63 All 63 (JSON)