mlsanityData to 7 Oct 2026

On the Nonlinearity of Layer Normalization

arXiv 2406.01255 ↗First posted 3 Jun 2024cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Layer normalization (LN) is a ubiquitous technique in deep learning but our theoretical understanding to it remains elusive. This paper investigates a new theoretical direction for LN, regarding to its nonlinearity and representation capacity. We investigate the representation capacity of a network with layerwise composition of linear and LN transformations, referred to as LN-Net. We theoretically show that, given m samples with any label assignment, an LN-Net with only 3 neurons in each layer and O(m) LN layers can correctly classify them. We further show the lower bound of the VC dimension of an LN-Net. The nonlinearity of LN can be amplified by group partition, which is also theoretically demonstrated with mild assumption and empirically supported by our experiments. Based on our analyses, we consider to design neural architecture by exploiting and amplifying the nonlinearity of LN, and the effectiveness is supported by our experiments.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

32 of 43 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023Segment Anythingcited by 8,158
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
20 more
  1. 2018Group Normalizationcited by 1,514
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  3. 2019Root Mean Square Layer Normalizationcited by 1,425
  4. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  5. 2020On Layer Normalization in the Transformer Architecturecited by 906
  6. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  7. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  8. 2019Understanding and Improving Layer Normalizationcited by 213
  9. 2018Understanding Batch Normalizationcited by 157
  10. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  11. 2020Minimum Width for Universal Approximationcited by 92
  12. 2020Normalization Techniques in Training DNNs: Methodology, Analysis and Applicationcited by 90
  13. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  14. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  15. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  16. 2018Towards Understanding Regularization in Batch Normalizationcited by 56
  17. 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 24
  18. 2019The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networkscited by 20
  19. 2020Group Whitening: Balancing Learning Efficiency and Representational Capacitycited by 13
  20. 2021Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependencecited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 3.

9 papers.

  1. 2025Transformers without Normalizationcited by 69
  2. 2025On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functionscited by 4
  3. 2024ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Modelscited by 3
  4. 2024AERO: Entropy-Guided Framework for Private LLM Inferencecited by 3
  5. 2025Parallel Layer Normalization for Universal Approximationcited by 0
  6. 2025On the Mathematical Impossibility of Safe Universal Approximatorscited by 0
  7. 2026A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Modelscited by 0
  8. 2026SaluNet: Enabling Total Plasticity in Normalization-Free Deep Networkscited by 0
  9. 2026Residual Learning in Empirical Asset Pricingcited by 0