# Quantifying Context Mixing in Transformers

arXiv: https://arxiv.org/abs/2301.12971 · HTML: https://mlsanity.com/paper/2301.12971/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 63 references that are in the map.
- First posted on arXiv 30 Jan 2023.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://mlsanity.com/paper/1911.03429/) (2019)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [Rationalizing Neural Predictions](https://mlsanity.com/paper/1606.04155/) (2016)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [Explaining by Removing: A Unified Framework for Model Explanation](https://mlsanity.com/paper/2011.14878/) (2020)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020)
- [The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective](https://mlsanity.com/paper/2202.01602/) (2022)
- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019)

## Built on by (most cited first, 30 of 43)

- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Inseq: An Interpretability Toolkit for Sequence Generation Models](https://mlsanity.com/paper/2302.13942/) (2023)
- [Attention Heads of Large Language Models: A Survey](https://mlsanity.com/paper/2409.03752/) (2024)
- [Identifying Semantic Induction Heads to Understand In-Context Learning](https://mlsanity.com/paper/2402.13055/) (2024)
- [Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era](https://mlsanity.com/paper/2403.08946/) (2024)
- [Explaining How Transformers Use Context to Build Predictions](https://mlsanity.com/paper/2305.12535/) (2023)
- [DecompX: Explaining Transformers Decisions by Propagating Token Decomposition](https://mlsanity.com/paper/2306.02873/) (2023)
- [What are you sinking? A geometric approach on attention sink](https://mlsanity.com/paper/2508.02546/) (2025)
- [Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.13663/) (2024)
- [Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization](https://mlsanity.com/paper/2510.13554/) (2025)
- [A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia](https://mlsanity.com/paper/2312.02073/) (2023)
- [DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformers](https://mlsanity.com/paper/2310.03686/) (2023)
- [Identifying and Adapting Transformer-Components Responsible for Gender Bias in an English Language Model](https://mlsanity.com/paper/2310.12611/) (2023)
- [Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations](https://mlsanity.com/paper/2504.12691/) (2025)
- [Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformers](https://mlsanity.com/paper/2310.09925/) (2023)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [Feature Interactions Reveal Linguistic Structure in Language Models](https://mlsanity.com/paper/2306.12181/) (2023)
- [Causal interventions expose implicit situation models for commonsense language understanding](https://mlsanity.com/paper/2306.03882/) (2023)
- [Quantifying the Plausibility of Context Reliance in Neural Machine Translation](https://mlsanity.com/paper/2310.01188/) (2023)
- [Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention](https://mlsanity.com/paper/2310.19084/) (2023)
- [Attention Speaks Volumes: Localizing and Mitigating Bias in Language Models](https://mlsanity.com/paper/2410.22517/) (2024)
- [SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generation](https://mlsanity.com/paper/2411.01710/) (2024)
- [NormXLogit: The Head-on-Top Never Lies](https://mlsanity.com/paper/2411.16252/) (2024)
- [Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Models](https://mlsanity.com/paper/2412.11187/) (2024)
- [Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement](https://mlsanity.com/paper/2505.23183/) (2025)
- [Counterfactuals As a Means for Evaluating Faithfulness of Attribution Methods in Autoregressive Language Models](https://mlsanity.com/paper/2408.11252/) (2024)
- [How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMs](https://mlsanity.com/paper/2606.10646/) (2026)
- [Contrast-CAT: Contrasting Activations for Enhanced Interpretability in Transformer-based Text Classifiers](https://mlsanity.com/paper/2507.21186/) (2025)
- [Cross-Attention is Half Explanation in Speech-to-Text Models](https://mlsanity.com/paper/2509.18010/) (2025)
- [Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translation](https://mlsanity.com/paper/2510.03115/) (2025)

## Abstract (arXiv)

Self-attention weights and their transformed variants have been the main source of information for analyzing token-to-token interactions in Transformer-based models. But despite their ease of interpretation, these weights are not faithful to the models' decisions as they are only one part of an encoder, and other components in the encoder layer can have considerable impact on information mixing in the output representations. In this work, by expanding the scope of analysis to the whole encoder block, we propose Value Zeroing, a novel context mixing score customized for Transformers that provides us with a deeper understanding of how information is mixed at each encoder layer. We demonstrate the superiority of our context mixing score over other analysis methods through a series of complementary evaluations with different viewpoints based on linguistically informed rationales, probing, and faithfulness analysis.