mlsanityData to 14 Oct 2026

On Layer Normalization in the Transformer Architecture

arXiv 2002.04745 ↗First posted 12 Feb 2020cs.LG cs.CL stat.MLcited by 922

Not on the map. Why?

Cited by 35 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyper-parameter tunings. In this paper, we first study theoretically why the learning rate warm-up stage is essential and show that the location of layer normalization matters. Specifically, we prove with mean field theory that at initialization, for the original-designed Post-LN Transformer, which places the layer normalization between the residual blocks, the expected gradients of the parameters near the output layer are large. Therefore, using a large learning rate on those gradients makes the training unstable. The warm-up stage is practically helpful for avoiding this problem. On the other hand, our theory also shows that if the layer normalization is put inside the residual blocks (recently proposed as Pre-LN Transformer), the gradients are well-behaved at initialization. This motivates us to remove the warm-up stage for the training of Pre-LN Transformers. We show in our experiments that Pre-LN Transformers without the warm-up stage can reach comparable results with baselines while requiring significantly less training time and hyper-parameter tuning on a wide range of applications.

Key facts. Cited by 922 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 84; 2022: 117; 2023: 134; 2024: 169; 2025: 179; 2026: 239.

922 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  3. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  5. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  6. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  7. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  8. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 614
  9. 2023Point Transformer V3: Simpler, Faster, Strongercited by 589
  10. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  11. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  12. 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 455
38 more
  1. 2021A Survey of Transformerscited by 401
  2. 2022Multimodal Learning with Transformers: A Surveycited by 400
  3. 2021Choose a Transformer: Fourier or Galerkincited by 343
  4. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  5. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 320
  6. 2023Customized Segment Anything Model for Medical Image Segmentationcited by 280
  7. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  8. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 260
  9. 2023From Sparse to Soft Mixtures of Expertscited by 248
  10. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  11. 2023TSMixer: An All-MLP Architecture for Time Series Forecastingcited by 240
  12. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 220
  13. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  14. 2022Continuous diffusion for categorical datacited by 214
  15. 2024CBraMod: A Criss-Cross Brain Foundation Model for EEG Decodingcited by 199
  16. 2022Pure Transformers are Powerful Graph Learnerscited by 188
  17. 2024Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendationscited by 188
  18. 2021StyTr2: Image Style Transfer with Transformerscited by 187
  19. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 187
  20. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  21. 2025Sundial: A Family of Highly Capable Time Series Foundation Modelscited by 180
  22. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  23. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 174
  24. 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 167
  25. 2022SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videoscited by 166
  26. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 164
  27. 2022Earthformer: Exploring Space-Time Transformers for Earth System Forecastingcited by 152
  28. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 147
  29. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  30. 2022NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphscited by 137
  31. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  32. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  33. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  34. 2022Latent Diffusion for Language Generationcited by 115
  35. 2021Global Self-Attention as a Replacement for Graph Convolutioncited by 114
  36. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  37. 2021Scaling Laws for Neural Machine Translationcited by 111
  38. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108

50 most cited of 922 All 922 (JSON)