mlsanityData to 7 Oct 2026

Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps

arXiv 2302.00456 ↗First posted 1 Feb 2023cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

Transformers are ubiquitous in wide tasks. Interpreting their internals is a pivotal goal. Nevertheless, their particular components, feed-forward (FF) blocks, have typically been less analyzed despite their substantial parameter amounts. We analyze the input contextualization effects of FF blocks by rendering them in the attention maps as a human-friendly visualization scheme. Our experiments with both masked- and causal-language models reveal that FF networks modify the input contextualization to emphasize specific types of linguistic compositions. In addition, FF and its surrounding components tend to cancel out each other's effects, suggesting potential redundancy in the processing of the Transformer layer.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 34 of its 66 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

34 of 66 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  11. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  12. 2020GLU Variants Improve Transformercited by 1,724
22 more
  1. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  2. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  3. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  4. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  5. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  6. 2020On Layer Normalization in the Transformer Architecturecited by 906
  7. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  8. 2019Are Sixteen Heads Really Better than One?cited by 803
  9. 2020Quantifying Attention Flow in Transformerscited by 759
  10. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  11. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
  12. 2019Revealing the Dark Secrets of BERTcited by 300
  13. 2020A Survey on the Explainability of Supervised Machine Learningcited by 225
  14. 2019On Identifiability in Transformerscited by 115
  15. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  16. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  17. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  18. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  19. 2021Incorporating Residual and Normalization Layers into Analysis of Masked Language Modelscited by 50
  20. 2022GlobEnc: Quantifying Global Token Attribution by Incorporating the Whole Encoder Layer in Transformerscited by 44
  21. 2020Positional Artefacts Propagate Through Masked Language Model Embeddingscited by 34
  22. 2022Entropy- and Distance-Based Predictors From GPT-2 Attention Patterns Predict Reading Times Over and Above GPT-2 Surprisalcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 12; 2026: 9.

33 papers.

Sort:

By area:

  1. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  2. 2024From Understanding to Utilization: A Survey on Explainability for Large Language Modelscited by 47
  3. 2024InternalInspector I2: Robust Confidence Estimation in LLMs through Internal Statescited by 29
  4. 2024Self-attention Networks Localize When QK-eigenspectrum Concentratescited by 19
  5. 2024Transformers Learn Low Sensitivity Functions: Investigations and Implicationscited by 13
  6. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  7. 2024Natural Language Processing RELIES on Linguisticscited by 9
  8. 2024Enhancing Multi-hop Reasoning through Knowledge Erasure in Large Language Model Editingcited by 9
  9. 2024Approximate Attributions for Off-the-Shelf Siamese Transformerscited by 7
  10. 2025Layerwise Importance Analysis of Feed-Forward Networks in Transformer-based Language Modelscited by 6
  11. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 6
  12. 2023Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Wordscited by 5
21 more
  1. 2026Large Vision-Language Models Get Lost in Attentioncited by 5
  2. 2024Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Modelscited by 4
  3. 2025Universality of Layer-Level Entropy-Weighted Quantization Beyond Model Architecture and Sizecited by 4
  4. 2026Subliminal Learning is a LoRA Artifactcited by 4
  5. 2025Dynamic Weight Grafting: Localizing Finetuned Factual Knowledge in Transformerscited by 3
  6. 2026Suppressing Final Layer Hidden State Jumps in Transformer Pretrainingcited by 3
  7. 2026NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networkscited by 3
  8. 2025LoKI: Low-damage Knowledge Implanting of Large Language Modelscited by 2
  9. 2026Quantifying LLM Attention-Head Stability: Implications for Circuit Universalitycited by 2
  10. 2024Efficient Concertormer for Image Deblurring and Beyondcited by 1
  11. 2024FFNet: MetaMixer-based Efficient Convolutional Mixer Designcited by 1
  12. 2025Merging Feed-Forward Sublayers for Compressed Transformerscited by 1
  13. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1
  14. 2025DePass: Unified Feature Attributing by Simple Decomposed Forward Passcited by 1
  15. 2025Order-Level Attention Similarity Across Language Models: A Latent Commonalitycited by 1
  16. 2026TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensorscited by 1
  17. 2026Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMscited by 1
  18. 2025BiHDTrans: binary hyperdimensional transformer for efficient multivariate time series classificationcited by 0
  19. 2025CorGi: Contribution-Guided Block-Wise Interval Caching for Training-Free Acceleration of Diffusion Transformerscited by 0
  20. 2026A Closer Look into LLMs for Table Understandingcited by 0
  21. 2026Layer-wise Target Propagation: Efficient Component Attribution through Target Centric Propagationcited by 0