mlsanityData to 14 Oct 2026

One Wide Feedforward is All You Need

arXiv 2309.01826 ↗First posted 4 Sep 2023cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

The Transformer architecture has two main non-embedding components: Attention and the Feed Forward Network (FFN). Attention captures interdependencies between words regardless of their position, while the FFN non-linearly transforms each input token independently. In this work we explore the role of the FFN, and find that despite taking up a significant fraction of the model's parameters, it is highly redundant. Concretely, we are able to substantially reduce the number of parameters with only a modest drop in accuracy by removing the FFN on the decoder layers and sharing a single FFN across the encoder. Finally we scale this architecture back to its original size by increasing the hidden dimension of the shared FFN, achieving substantial gains in both accuracy and latency with respect to the original Transformer Big.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 35 of its 42 references that are in the map. First posted on arXiv 4 Sep 2023.

Builds on

35 of 42 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2016Layer Normalizationcited by 5,669
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  11. 2020Extracting Training Data from Large Language Modelscited by 2,189
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
23 more
  1. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  3. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  4. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  7. 2019Are Sixteen Heads Really Better than One?cited by 816
  8. 2018Universal Transformerscited by 647
  9. 2016Using the Output Embedding to Improve Language Modelscited by 272
  10. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
  11. 2018Scaling Neural Machine Translationcited by 204
  12. 2021Scaling Laws for Neural Machine Translationcited by 111
  13. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 105
  14. 2019Getting Gender Right in Neural Machine Translationcited by 98
  15. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  16. 2016Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Changecited by 63
  17. 2019On Measuring Gender Bias in Translation of Gender-neutral Pronounscited by 60
  18. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  19. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 43
  20. 2019Embedding Comparator: Visualizing Differences in Global Structure and Local Neighborhoods via Small Multiplescited by 38
  21. 2022EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generationcited by 21
  22. 2023Learning Language-Specific Layers for Multilingual Machine Translationcited by 13
  23. 2022Non-Autoregressive Neural Machine Translation: A Call for Claritycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 7; 2026: 2.

15 papers.

Sort:

By area:

  1. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuningcited by 64
  2. 2025RelaCtrl: Relevance-Guided Efficient Control for Diffusion Transformerscited by 18
  3. 2024Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Visioncited by 13
  4. 2024Cross-layer Attention Sharing for Pre-trained Large Language Modelscited by 5
  5. 2024Mitigating Sycophancy in Decoder-Only Transformer Architectures: Synthetic Data Interventioncited by 5
  6. 2025Attention-Only Transformers via Unrolled Subspace Denoisingcited by 5
  7. 2025Universality of Layer-Level Entropy-Weighted Quantization Beyond Model Architecture and Sizecited by 4
  8. 2025Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Modelscited by 4
  9. 2024Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspectivecited by 3
  10. 2024Activation Sparsity Opportunities for Compressing General Large Language Modelscited by 1
  11. 2025Merging Feed-Forward Sublayers for Compressed Transformerscited by 1
  12. 2025CascadedViT: Cascaded Chunk-FeedForward and Cascaded Group Attention Vision Transformercited by 0
3 more
  1. 2025Efficient-Husformer: Efficient Multimodal Transformer Hyperparameter Optimization for Stress and Cognitive Loadscited by 0
  2. 2026Exploring the Limits of Pruning: Task-Specific Neurons, Model Collapse, and Recovery in Task-Specific Large Language Modelscited by 0
  3. 2026More Than Memory: Task-Conditioned Signed FFN Writes in Long-Context Retrievalcited by 0