mlsanityData to 14 Oct 2026

Investigating the Role of Feed-Forward Networks in Transformers Using Parallel Attention and Feed-Forward Net Design

arXiv 2305.13297 ↗First posted 22 May 2023cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper investigates the key role of Feed-Forward Networks (FFNs) in transformer models by utilizing the Parallel Attention and Feed-Forward Net Design (PAF) architecture, and comparing it to their Series Attention and Feed-Forward Net Design (SAF) counterparts. Central to the effectiveness of PAF are two main assumptions regarding the FFN block and the attention block within a layer: 1) the primary function of the FFN block is to maintain isotropy among token embeddings and prevent their degeneration, and 2) the residual norm computed in the attention block is substantially smaller than the input token embedding norm. To empirically validate these assumptions, we train PAF variants of two large language models (RoBERTa-large and bert-large-uncased). Our results demonstrate that both assumptions hold true in the PAF design. This study contributes to a deeper understanding of the roles and interactions between FFNs and self-attention mechanisms in transformer architectures.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 17 of its 23 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

17 of 23 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  10. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  11. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  12. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
5 more
  1. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  2. 2018Universal Language Model Fine-tuning for Text Classificationcited by 52
  3. 2020Very Deep Transformers for Neural Machine Translationcited by 50
  4. 2020DeFormer: Decomposing Pre-trained Transformers for Faster Question Answeringcited by 29
  5. 2020Pretrained Transformers for Simple Question Answering over Knowledge Graphscited by 12

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025POI-Enhancer: An LLM-based Semantic Enhancement Framework for POI Representation Learningcited by 10
  2. 2025MoFE: Mixture of Frozen Experts Architecturecited by 1
  3. 2025Bi-directional Recurrence Improves Transformer in Partially Observable Markov Decision Processescited by 0
  4. 2025Resting Neurons, Active Insights: Robustifying Activation Sparsity in LLMs via Spontaneitycited by 0