mlsanityData to 7 Oct 2026

On the impact of activation and normalization in obtaining isometric embeddings at initialization

arXiv 2305.18399 ↗First posted 28 May 2023cs.LG cs.AI stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

In this paper, we explore the structure of the penultimate Gram matrix in deep neural networks, which contains the pairwise inner products of outputs corresponding to a batch of inputs. In several architectures it has been observed that this Gram matrix becomes degenerate with depth at initialization, which dramatically slows training. Normalization layers, such as batch or layer normalization, play a pivotal role in preventing the rank collapse issue. Despite promising advances, the existing theoretical results do not extend to layer normalization, which is widely used in transformers, and can not quantitatively characterize the role of non-linear activations. To bridge this gap, we prove that layer normalization, in conjunction with activation layers, biases the Gram matrix of a multilayer perceptron towards the identity matrix at an exponential rate with depth at initialization. We quantify this rate using the Hermite expansion of the activation function.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 33 of its 42 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

33 of 42 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Layer Normalizationcited by 5,579
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  9. 2020On Layer Normalization in the Transformer Architecturecited by 906
  10. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  11. 2017Self-Normalizing Neural Networkscited by 591
  12. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
21 more
  1. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  2. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  3. 2016Deep Information Propagationcited by 194
  4. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  5. 2018Understanding Batch Normalizationcited by 157
  6. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  7. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  8. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  9. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  10. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  11. 2019A Mean Field Theory of Batch Normalizationcited by 58
  12. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
  13. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
  14. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  15. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  16. 2021Batch Normalization Orthogonalizes Representations in Deep Random Networkscited by 28
  17. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  18. 2018Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimizationcited by 27
  19. 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 24
  20. 2023Efficient displacement convex optimization with particle gradient descentcited by 5
  21. 2022On Bridging the Gap between Mean Field and Finite Width in Deep Random Neural Networks with Batch Normalizationcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 8; 2025: 2.

13 papers.

Sort:

  1. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  2. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 45
  3. 2024Measure-to-measure interpolation using Transformerscited by 28
  4. 2024Understanding and Minimising Outlier Features in Neural Network Trainingcited by 10
  5. 2023Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosioncited by 8
  6. 2024Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamicscited by 7
  7. 2025Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticitycited by 5
  8. 2024ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Modelscited by 3
  9. 2024AERO: Entropy-Guided Framework for Private LLM Inferencecited by 3
  10. 2023Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamicscited by 2
  11. 2024Provable optimal transport with transformers: The essence of depth and prompt engineeringcited by 1
  12. 2024Emergence of Globally Attracting Fixed Points in Deep Neural Networks With Nonlinear Activationscited by 1
1 more
  1. 2025IBNorm: Information-Bottleneck Inspired Normalization for Representation Learningcited by 1