# On the Expressivity Role of LayerNorm in Transformers' Attention

arXiv: https://arxiv.org/abs/2305.02582 · HTML: https://mlsanity.com/paper/2305.02582/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 4 May 2023.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Understanding and Improving Layer Normalization](https://mlsanity.com/paper/1911.07013/) (2019)
- [Stolen Probability: A Structural Weakness of Neural Language Models](https://mlsanity.com/paper/2005.02433/) (2020)
- [Low-Rank Softmax Can Have Unargmaxable Classes in Theory but Rarely in Practice](https://mlsanity.com/paper/2203.06462/) (2022)

## Built on by (most cited first, 30 of 33)

- [Universal Neurons in GPT2 Language Models](https://mlsanity.com/paper/2401.12181/) (2024)
- [On the Role of Attention Masks and LayerNorm in Transformers](https://mlsanity.com/paper/2405.18781/) (2024)
- [Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformers](https://mlsanity.com/paper/2404.04393/) (2024)
- [Prompting a Pretrained Transformer Can Be a Universal Approximator](https://mlsanity.com/paper/2402.14753/) (2024)
- [Lifelong Knowledge Editing requires Better Regularization](https://mlsanity.com/paper/2502.01636/) (2025)
- [A Generalizable Deep Learning System for Cardiac MRI](https://mlsanity.com/paper/2312.00357/) (2023)
- [Traveling Words: A Geometric Interpretation of Transformers](https://mlsanity.com/paper/2309.07315/) (2023)
- [Transformer Normalisation Layers and the Independence of Semantic Subspaces](https://mlsanity.com/paper/2406.17837/) (2024)
- [Embedding Geometries of Contrastive Language-Image Pre-Training](https://mlsanity.com/paper/2409.13079/) (2024)
- [The Transformer Cookbook](https://mlsanity.com/paper/2510.00368/) (2025)
- [ZeroS: Zero-Sum Linear Attention for Efficient Transformers](https://mlsanity.com/paper/2602.05230/) (2026)
- [Logicbreaks: A Framework for Understanding Subversion of Rule-based Inference](https://mlsanity.com/paper/2407.00075/) (2024)
- [Lambda-Skip Connections: the architectural component that prevents Rank Collapse](https://mlsanity.com/paper/2410.10609/) (2024)
- [Grappa -- A Machine Learned Molecular Mechanics Force Field](https://mlsanity.com/paper/2404.00050/) (2024)
- [Stealth edits to large language models](https://mlsanity.com/paper/2406.12670/) (2024)
- [stEnTrans: Transformer-based deep learning for spatial transcriptomics enhancement](https://mlsanity.com/paper/2407.08224/) (2024)
- [AERO: Entropy-Guided Framework for Private LLM Inference](https://mlsanity.com/paper/2410.13060/) (2024)
- [Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective](https://mlsanity.com/paper/2411.03033/) (2024)
- [Understanding the Role of Layer Normalization in Label-Skewed Federated Learning](https://mlsanity.com/paper/2308.09565/) (2023)
- [Geometry and Dynamics of LayerNorm](https://mlsanity.com/paper/2405.04134/) (2024)
- [UnitNorm: Rethinking Normalization for Transformers in Time Series](https://mlsanity.com/paper/2405.15903/) (2024)
- [Observable Propagation: Uncovering Feature Vectors in Transformers](https://mlsanity.com/paper/2312.16291/) (2023)
- [Self-consistent context aware conformer transducer for speech recognition](https://mlsanity.com/paper/2402.06592/) (2024)
- [Adaptive Engram Memory System for Indonesian Language Model: Generative AI Based on TOBA LM for Batak and Minang Language](https://mlsanity.com/paper/2603.10006/) (2026)
- [The Scaling Properties of Implicit Deductive Reasoning in Transformers](https://mlsanity.com/paper/2605.04330/) (2026)
- [SLaNC: Static LayerNorm Calibration](https://mlsanity.com/paper/2410.10553/) (2024)
- [Leveraging Teleconnections with Physics-Informed Graph Attention Networks for Long-Range Extreme Rainfall Forecasting in Thailand](https://mlsanity.com/paper/2510.12328/) (2025)
- [Stop Jostling: Adaptive Negative Sampling Reduces the Marginalization of Low-Resource Language Tokens by Cross-Entropy Loss](https://mlsanity.com/paper/2601.22439/) (2026)
- [Enhanced QKNorm normalization for neural transformers with the Lp norm](https://mlsanity.com/paper/2602.05006/) (2026)
- [Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling](https://mlsanity.com/paper/2604.19147/) (2026)

## Abstract (arXiv)

Layer Normalization (LayerNorm) is an inherent component in all Transformer-based models. In this paper, we show that LayerNorm is crucial to the expressivity of the multi-head attention layer that follows it. This is in contrast to the common belief that LayerNorm's only role is to normalize the activations during the forward pass, and their gradients during the backward pass. We consider a geometric interpretation of LayerNorm and show that it consists of two components: (a) projection of the input vectors to a d-1 space that is orthogonal to the left[1,1,...,1right] vector, and (b) scaling of all vectors to the same norm of √d. We show that each of these components is important for the attention layer that follows it in Transformers: (a) projection allows the attention mechanism to create an attention query that attends to all keys equally, offloading the need to learn this operation by the attention; and (b) scaling allows each key to potentially receive the highest attention, and prevents keys from being "un-select-able". We show empirically that Transformers do indeed benefit from these properties of LayeNorm in general language modeling and even in computing simple functions such as "majority". Our code is available at https://github.com/tech-srl/layernormexpressivityrole .