mlsanityData to 7 Oct 2026

NormFormer: Improved Transformer Pretraining with Extra Normalization

arXiv 2110.09456 ↗First posted 18 Oct 2021cs.CL cs.AIcited by 83

Not on the map. Why?

Abstract (arXiv)

During pretraining, the Pre-LayerNorm transformer suffers from a gradient magnitude mismatch: gradients at early layers are much larger than at later layers. These issues can be alleviated by our proposed NormFormer architecture, which adds three normalization operations to each layer: a Layer Norm after self attention, head-wise scaling of self-attention outputs, and a Layer Norm after the first fully connected layer. The extra operations incur negligible compute cost (+0.4% parameter increase), but improve pretraining perplexity and downstream task performance for both causal and masked language models ranging from 125 Million to 2.7 Billion parameters. For example, adding NormFormer on top of our strongest 1.3B parameter baseline can reach equal perplexity 24% faster, or converge 0.27 perplexity better in the same compute budget. This model reaches GPT3-Large (1.3B) zero shot performance 60% faster. For masked language modeling, NormFormer improves fine-tuned GLUE performance by 1.9% on average. Code to train NormFormer models is available in fairseq https://github.com/pytorch/fairseq/tree/main/examples/normformer .

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 29 of its 36 references that are in the map. First posted on arXiv 18 Oct 2021.

Builds on

29 of 36 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
17 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2020GLU Variants Improve Transformercited by 1,724
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  4. 2017Mixed Precision Trainingcited by 1,135
  5. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  6. 2020On Layer Normalization in the Transformer Architecturecited by 906
  7. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  8. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  9. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  10. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  11. 2020Understanding the Difficulty of Training Transformerscited by 210
  12. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  13. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 156
  14. 2020EarlyBERT: Efficient BERT Training via Early-bird Lottery Ticketscited by 79
  15. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  16. 2019Improving Transformer Models by Reordering their Sublayerscited by 52
  17. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 20; 2024: 23; 2025: 11; 2026: 15.

83 papers.

Sort:

By area:

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  2. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  3. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  6. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  7. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  8. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  9. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  10. 2022MetaFormer Baselines for Visioncited by 131
  11. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  12. 2022Particle Transformer for Jet Taggingcited by 95
38 more
  1. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  2. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  3. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
  4. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  5. 2024Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Modelscited by 44
  6. 2024OmniJet-α: The first cross-task foundation model for particle physicscited by 41
  7. 2024LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexitycited by 41
  8. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  9. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  10. 2025Understanding Transformer from the Perspective of Associative Memorycited by 31
  11. 2023PC-JeDi: Diffusion for Particle Cloud Generation in High Energy Physicscited by 27
  12. 2023NorBench -- A Benchmark for Norwegian Language Modelscited by 23
  13. 2023FFT-based Dynamic Token Mixer for Visioncited by 22
  14. 2023TorchDEQ: A Library for Deep Equilibrium Modelscited by 21
  15. 2022METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signalscited by 20
  16. 2024Is Tokenization Needed for Masked Particle Modelling?cited by 20
  17. 2024Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMscited by 20
  18. 2022Foundation Transformerscited by 19
  19. 2025Differentially Private Steering for Large Language Model Alignmentcited by 15
  20. 2023SPANet: Frequency-balancing Token Mixer using Spectral Pooling Aggregation Modulationcited by 14
  21. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
  22. 2023Dual PatchNormcited by 9
  23. 2022Cross-lingual Similarity of Multilingual Representations Revisitedcited by 8
  24. 2024AntLM: Bridging Causal and Masked Language Modelscited by 8
  25. 2025Residual Matrix Transformers: Scaling the Size of the Residual Streamcited by 8
  26. 2023BranchNorm: Robustly Scaling Extremely Deep Transformerscited by 6
  27. 2024Transformer Normalisation Layers and the Independence of Semantic Subspacescited by 6
  28. 2024Stable Language Model Pre-training by Reducing Embedding Variabilitycited by 6
  29. 2022Convolutional Xformers for Visioncited by 5
  30. 2024The Credibility Transformercited by 5
  31. 2025Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Modelscited by 5
  32. 2026Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Modelscited by 5
  33. 2022ParCNetV2: Oversized Kernel with Enhanced Attentioncited by 4
  34. 2023On the Weight Dynamics of Deep Normalized Networkscited by 4
  35. 2023Tokenization with Factorized Subword Encodingcited by 4
  36. 2024A Survey of Vision Transformers in Autonomous Driving: Current Trends and Future Directionscited by 4
  37. 2024LayerNorm: A key component in parameter-efficient fine-tuningcited by 4
  38. 2025Human-Level Competitive Pok'emon via Scalable Offline Reinforcement Learning with Transformerscited by 4

50 most cited of 83 All 83 (JSON)