mlsanityData to 7 Oct 2026

Layer Normalization

arXiv 1607.06450 ↗First posted 21 Jul 2016stat.ML cs.LGcited by 5,579

Not on the map. Why?

Cited by 209 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute a mean and variance which are then used to normalize the summed input to that neuron on each training case. This significantly reduces the training time in feed-forward neural networks. However, the effect of batch normalization is dependent on the mini-batch size and it is not obvious how to apply it to recurrent neural networks. In this paper, we transpose batch normalization into layer normalization by computing the mean and variance used for normalization from all of the summed inputs to the neurons in a layer on a single training case. Like batch normalization, we also give each neuron its own adaptive bias and gain which are applied after the normalization but before the non-linearity. Unlike batch normalization, layer normalization performs exactly the same computation at training and test times. It is also straightforward to apply to recurrent neural networks by computing the normalization statistics separately at each time step. Layer normalization is very effective at stabilizing the hidden state dynamics in recurrent networks. Empirically, we show that layer normalization can substantially reduce the training time compared with previously published techniques.

Key facts. Cited by 5,579 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jul 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 4; 2019: 3; 2020: 3; 2021: 821; 2022: 968; 2023: 1119; 2024: 1125; 2025: 863; 2026: 673.

5,579 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Segment Anythingcited by 8,158
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  11. 2022Visual Prompt Tuningcited by 1,674
  12. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,618
38 more
  1. 2021ViViT: A Video Vision Transformercited by 1,610
  2. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  7. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  8. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  9. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2023Mastering Diverse Domains through World Modelscited by 1,193
  13. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  14. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  15. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  16. 2021Transformers in Vision: A Surveycited by 1,009
  17. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 954
  18. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  19. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 906
  20. 2021Multiscale Vision Transformerscited by 896
  21. 2021Uformer: A General U-Shaped Transformer for Image Restorationcited by 864
  22. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  23. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  24. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  25. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  26. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  27. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  28. 2022A Generalist Agentcited by 802
  29. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  30. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  31. 2021Domain Generalization: A Surveycited by 760
  32. 2021Going deeper with Image Transformerscited by 759
  33. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  34. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  35. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  36. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  37. 2022Simple Baselines for Image Restorationcited by 701
  38. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691

50 most cited of 5,579 All 5,579 (JSON)