mlsanityData to 7 Oct 2026

Incorporating Residual and Normalization Layers into Analysis of Masked Language Models

arXiv 2109.07152 ↗First posted 15 Sep 2021cs.CLcited by 50

Not on the map. Why?

Abstract (arXiv)

Transformer architecture has become ubiquitous in the natural language processing field. To interpret the Transformer-based models, their attention patterns have been extensively analyzed. However, the Transformer architecture is not only composed of the multi-head attention; other components can also contribute to Transformers' progressive performance. In this study, we extended the scope of the analysis of Transformers from solely the attention patterns to the whole attention block, i.e., multi-head attention, residual connection, and layer normalization. Our analysis of Transformer-based masked language models shows that the token-to-token interaction performed via attention has less impact on the intermediate representations than previously assumed. These results provide new intuitive explanations of existing reports; for example, discarding the learned attention patterns tends not to adversely affect the performance. The codes of our experiments are publicly available.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 34 of its 47 references that are in the map. First posted on arXiv 15 Sep 2021.

Builds on

34 of 47 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2016Layer Normalizationcited by 5,579
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  11. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  12. 2020On Layer Normalization in the Transformer Architecturecited by 906
22 more
  1. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  2. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  3. 2019Are Sixteen Heads Really Better than One?cited by 803
  4. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  5. 2020Quantifying Attention Flow in Transformerscited by 759
  6. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  7. 2019Revealing the Dark Secrets of BERTcited by 300
  8. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  9. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  10. 2015Training Very Deep Networkscited by 172
  11. 2019Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)cited by 128
  12. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  13. 2019On Identifiability in Transformerscited by 115
  14. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  15. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 87
  16. 2020Human Sentence Processing: Recurrence or Attention?cited by 51
  17. 2020Word Rotator's Distancecited by 32
  18. 2018OpenNMT: Neural Machine Translation Toolkitcited by 27
  19. 2019From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentionscited by 22
  20. 2021Rethinking Skip Connection with Layer Normalization in Transformers and ResNetscited by 22
  21. 2015Measuring Word Significance using Distributed Representations of Wordscited by 21
  22. 2018An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 11; 2024: 14; 2025: 10; 2026: 7.

50 papers.

Sort:

By area:

  1. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  2. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 100
  3. 2023Inseq: An Interpretability Toolkit for Sequence Generation Modelscited by 78
  4. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  5. 2022Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformercited by 45
  6. 2022GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformerscited by 44
  7. 2023Quantifying Context Mixing in Transformerscited by 43
  8. 2023Explaining How Transformers Use Context to Build Predictionscited by 42
  9. 2023Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Mapscited by 33
  10. 2023DecompX: Explaining Transformers Decisions by Propagating Token Decompositioncited by 31
  11. 2022ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Modelscited by 25
  12. 2022Entropy- and Distance-Based Predictors From GPT-2 Attention Patterns Predict Reading Times Over and Above GPT-2 Surprisalcited by 14
38 more
  1. 2023Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformerscited by 13
  2. 2024An Unsupervised Approach to Achieve Supervised-Level Explainability in Healthcare Recordscited by 12
  3. 2024Evaluating Human Alignment and Model Faithfulness of LLM Rationalecited by 12
  4. 2025A Close Look at Decomposition-based XAI-Methods for Transformer Language Modelscited by 10
  5. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  6. 2023Transformer Language Models Handle Word Frequency in Prediction Headcited by 8
  7. 2024Better Explain Transformers by Illuminating Important Informationcited by 8
  8. 2023Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attentioncited by 7
  9. 2025Are We Paying Attention to Her? Investigating Gender Disambiguation and Attention in Machine Translationcited by 7
  10. 2023Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictionscited by 6
  11. 2024Transformer Normalisation Layers and the Independence of Semantic Subspacescited by 6
  12. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 6
  13. 2022On the Locality of Attention in Direct Speech Translationcited by 5
  14. 2024Order-Independence Without Fine Tuningcited by 5
  15. 2024LibraGrad: Balancing Gradient Flow for Universally Better Vision Transformer Attributionscited by 5
  16. 2022Life after BERT: What do Other Muppets Understand about Language?cited by 4
  17. 2024NormXLogit: The Head-on-Top Never Liescited by 4
  18. 2024Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Modelscited by 4
  19. 2024Counterfactuals As a Means for Evaluating Faithfulness of Attribution Methods in Autoregressive Language Modelscited by 3
  20. 2025LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Modelscited by 3
  21. 2025Generalized Attention Flow: Feature Attribution for Transformer Models via Maximum Flowcited by 3
  22. 2025SEReDeEP: Hallucination Detection in Retrieval-Augmented Models via Semantic Entropy and Context-Parameter Fusioncited by 3
  23. 2023Computational modeling of semantic changecited by 2
  24. 2023SpeechAlign: a Framework for Speech Translation Alignment Evaluationcited by 2
  25. 2024UnitNorm: Rethinking Normalization for Transformers in Time Seriescited by 2
  26. 2024Explanation Regularisation through the Lens of Attributionscited by 1
  27. 2024How Language Models Prioritize Contextual Grammatical Cues?cited by 1
  28. 2024On Explaining with Attention Matricescited by 1
  29. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1
  30. 2025Order-Level Attention Similarity Across Language Models: A Latent Commonalitycited by 1
  31. 2026TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensorscited by 1
  32. 2026Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarkscited by 1
  33. 2025CorGi: Contribution-Guided Block-Wise Interval Caching for Training-Free Acceleration of Diffusion Transformerscited by 0
  34. 2026Layer-wise Target Propagation: Efficient Component Attribution through Target Centric Propagationcited by 0
  35. 2026Positional Encodings Anchor Spatial Structure in Vision Transformers: A Geometric Perspective on Robustnesscited by 0
  36. 2026Contribution Weights: A Geometrical Analysis of Self-Attention Transformerscited by 0
  37. 2026Residual Dominance as a Structural Account of Last-Item Reliance in Causal Self-Attention Recommenderscited by 0
  38. 2026Scaling Attention Head Analysis via Gradient-Based Attribution in Context-Aware Machine Translationcited by 0