Data to 14 Oct 2026

Understanding and Improving Layer Normalization

arXiv 1911.07013 ↗First posted 16 Nov 2019cs.LG cs.CL stat.MLcited by 217

Not on the map. Why?

Abstract (arXiv)

Layer normalization (LayerNorm) is a technique to normalize the distributions of intermediate layers. It enables smoother gradients, faster training, and better generalization accuracy. However, it is still unclear where the effectiveness stems from. In this paper, our main contribution is to take a step further in understanding LayerNorm. Many of previous studies believe that the success of LayerNorm comes from forward normalization. Unlike them, we find that the derivatives of the mean and variance are more important than forward normalization by re-centering and re-scaling backward gradients. Furthermore, we find that the parameters of LayerNorm, including the bias and gain, increase the risk of over-fitting and do not work in most cases. Experiments show that a simple version of LayerNorm (LayerNorm-simple) without the bias and gain outperforms LayerNorm on four datasets. It obtains the state-of-the-art performance on En-Vi machine translation. To address the over-fitting problem, we propose a new normalization method, Adaptive Normalization (AdaNorm), by replacing the bias and gain with a new transformation function. Experiments show that AdaNorm demonstrates better results than LayerNorm on seven out of eight datasets.

Key facts. Cited by 217 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 23; 2023: 30; 2024: 37; 2025: 69; 2026: 43.

217 papers.

Sort:

By area:

2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
2021A Survey of Transformerscited by 401
2024Navigation World Modelscited by 272
2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 252
2025WorldMem: Long-term Consistent World Simulation with Memorycited by 168
2026Advancing Open-source World Modelscited by 154
2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 86
2021Adversarially Adaptive Normalization for Single Domain Generalizationcited by 85
2025Transformers without Normalizationcited by 70

38 more

2025WMPO: World Model-based Policy Optimization for Vision-Language-Action Modelscited by 70
2024SINGAPO: Single Image Controlled Generation of Articulated Parts in Objectscited by 69
2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuningcited by 64
2025HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animationcited by 59
2022THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryptioncited by 57
2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 55
2021A Survey on Green Deep Learningcited by 52
2023CAGE: Controllable Articulation GEnerationcited by 52
2025ConceptAttention: Diffusion Transformers Learn Highly Interpretable Featurescited by 52
2026Infinite Worlds with Versatile Interactionscited by 48
2024Neural Redshift: Random Networks are not Random Functionscited by 43
2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 43
2025What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?cited by 43
2022Continual Normalization: Rethinking Batch Normalization for Online Continual Learningcited by 40
2023Overcoming Data Scarcity in Biomedical Imaging with a Foundational Multi-Task Modelcited by 40
2023SODFormer: Streaming Object Detection with Transformer Using Events and Framescited by 39
2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 36
2025MoVieS: Motion-Aware 4D Dynamic View Synthesis in One Secondcited by 35
2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
20245%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Taskscited by 34
2023On the Expressivity Role of LayerNorm in Transformers' Attentioncited by 33
2024GraphKAN: Enhancing Feature Extraction with Graph Kolmogorov Arnold Networkscited by 32
2025HiF-VLA: Hindsight, Insight and Foresight through Motion Representation for Vision-Language-Action Modelscited by 32
2024TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Modelscited by 31
2023Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformerscited by 30
2023Sequence Length Independent Norm-Based Generalization Bounds for Transformerscited by 29
2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 27
2024Cross-conditioned Diffusion Model for Medical Image to Image Translationcited by 26
2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 25
2023Co-Evolution of Pose and Mesh for 3D Human Body Estimation from Videocited by 25
2024Streaming Deep Reinforcement Learning Finally Workscited by 25
2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
2023HumanLiff: Layer-wise 3D Human Generation with Diffusion Modelcited by 23
2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 23
2021Rethinking Skip Connection with Layer Normalization in Transformers and ResNetscited by 22
2023Modality-Agnostic Variational Compression of Implicit Neural Representationscited by 22
2024Privacy Backdoors: Stealing Data with Corrupted Pretrained Modelscited by 22

50 most cited of 217 All 217 (JSON)