mlsanityData to 7 Oct 2026

Quantifying Context Mixing in Transformers

arXiv 2301.12971 ↗First posted 30 Jan 2023cs.CL cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

Self-attention weights and their transformed variants have been the main source of information for analyzing token-to-token interactions in Transformer-based models. But despite their ease of interpretation, these weights are not faithful to the models' decisions as they are only one part of an encoder, and other components in the encoder layer can have considerable impact on information mixing in the output representations. In this work, by expanding the scope of analysis to the whole encoder block, we propose Value Zeroing, a novel context mixing score customized for Transformers that provides us with a deeper understanding of how information is mixed at each encoder layer. We demonstrate the superiority of our context mixing score over other analysis methods through a series of complementary evaluations with different viewpoints based on linguistically informed rationales, probing, and faithfulness analysis.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 49 of its 63 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

49 of 63 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  10. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  11. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  12. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
37 more
  1. 2019Attention is not Explanationcited by 916
  2. 2020Quantifying Attention Flow in Transformerscited by 759
  3. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  4. 2019Attention is not not Explanationcited by 629
  5. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  6. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  7. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  8. 2019Revealing the Dark Secrets of BERTcited by 300
  9. 2016Rationalizing Neural Predictionscited by 281
  10. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  11. 2020Explaining by Removing: A Unified Framework for Model Explanationcited by 218
  12. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  13. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  14. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 158
  15. 2022The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspectivecited by 156
  16. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  17. 2019Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)cited by 128
  18. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  19. 2019On Identifiability in Transformerscited by 115
  20. 2018Context-Aware Neural Machine Translation Learns Anaphora Resolutioncited by 97
  21. 2020Evaluating Explanations: How much do explanations from the teacher aid students?cited by 94
  22. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 87
  23. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  24. 2017The Promise and Peril of Human Evaluation for Model Interpretabilitycited by 52
  25. 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 50
  26. 2019When and Why is Document-level Context Useful in Neural Machine Translation?cited by 45
  27. 2022GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformerscited by 44
  28. 2022PERFECT: Prompt-free and Efficient Few-shot Learning with Language Modelscited by 40
  29. 2019Do Human Rationales Improve Machine Explanations?cited by 33
  30. 2020Word Rotator's Distancecited by 32
  31. 2019Fine-grained Sentiment Analysis with Faithful Attentioncited by 23
  32. 2021Rethinking Skip Connection with Layer Normalization in Transformers and ResNetscited by 22
  33. 2022How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformerscited by 20
  34. 2021Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representationscited by 16
  35. 2020Telling BERT's full story: from Local Attention to Global Aggregationcited by 15
  36. 2022A Fine-grained Interpretability Evaluation Benchmark for Neural NLPcited by 15
  37. 2022A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processingcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 10; 2025: 11; 2026: 10.

43 papers.

Sort:

By area:

  1. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  2. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  3. 2024Attention Heads of Large Language Models: A Surveycited by 65
  4. 2024Identifying Semantic Induction Heads to Understand In-Context Learningcited by 46
  5. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  6. 2023Explaining How Transformers Use Context to Build Predictionscited by 42
  7. 2023DecompX: Explaining Transformers Decisions by Propagating Token Decompositioncited by 31
  8. 2025What are you sinking? A geometric approach on attention sinkcited by 26
  9. 2024Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generationcited by 25
  10. 2023A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepediacited by 24
  11. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 23
  12. 2023DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformerscited by 22
31 more
  1. 2023Identifying and Adapting Transformer-Components Responsible for Gender Bias in an English Language Modelcited by 22
  2. 2025Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associationscited by 14
  3. 2023Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformerscited by 13
  4. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  5. 2023Feature Interactions Reveal Linguistic Structure in Language Modelscited by 8
  6. 2023Causal interventions expose implicit situation models for commonsense language understandingcited by 7
  7. 2023Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attentioncited by 7
  8. 2023Quantifying the Plausibility of Context Reliance in Neural Machine Translationcited by 6
  9. 2024Attention Speaks Volumes: Localizing and Mitigating Bias in Language Modelscited by 6
  10. 2024SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generationcited by 6
  11. 2024NormXLogit: The Head-on-Top Never Liescited by 4
  12. 2024Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Modelscited by 4
  13. 2025Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreementcited by 4
  14. 2024Counterfactuals As a Means for Evaluating Faithfulness of Attribution Methods in Autoregressive Language Modelscited by 3
  15. 2026How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMscited by 3
  16. 2025Contrast-CAT: Contrasting Activations for Enhanced Interpretability in Transformer-based Text Classifierscited by 1
  17. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1
  18. 2025Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translationcited by 1
  19. 2025DePass: Unified Feature Attributing by Simple Decomposed Forward Passcited by 1
  20. 2025Enhancing Long Document Long Form Summarisation with Self-Planningcited by 1
  21. 2026Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architecturescited by 1
  22. 2024Transformer with Controlled Attention for Synchronous Motion Captioningcited by 0
  23. 2025Activation Transport Operatorscited by 0
  24. 2026Faithfulness Evaluation for Decoder-only LLM Attributions with Controlled Retained Informationcited by 0
  25. 2026Evaluating Explainable AI Attribution Methods in Neural Machine Translation via Attention-Guided Knowledge Distillationcited by 0
  26. 2026Multi-component Causal Tracing in Large Language Modelscited by 0
  27. 2026When Context Misleads: Surprisal, Energy and Attention Entropy as Metrics of Coherence Illusions in LLMscited by 0
  28. 2026Warranted Attention: Learning What to Pass from Attention to Predictioncited by 0
  29. 2026Scaling Attention Head Analysis via Gradient-Based Attribution in Context-Aware Machine Translationcited by 0
  30. 2026Decoupling Token Roles in Autoregressive Pretrainingcited by 0
  31. 2026Late Attention Layers Alone Can Copy Entity Tokens, but Not Without Attending to Their Contextcited by 0