# Geometry and Dynamics of LayerNorm

arXiv: https://arxiv.org/abs/2405.04134 · HTML: https://mlsanity.com/paper/2405.04134/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 6 references that are in the map.
- First posted on arXiv 7 May 2024.

## Builds on (most cited first, 4 of 4)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [On the Expressivity Role of LayerNorm in Transformers' Attention](https://mlsanity.com/paper/2305.02582/) (2023)
- [Traveling Words: A Geometric Interpretation of Transformers](https://mlsanity.com/paper/2309.07315/) (2023)

## Built on by (most cited first, 2 of 2)

- [Chebyshev Polynomial-Based Kolmogorov-Arnold Networks: An Efficient Architecture for Nonlinear Function Approximation](https://mlsanity.com/paper/2405.07200/) (2024)
- [Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale](https://mlsanity.com/paper/2606.19491/) (2026)

## Abstract (arXiv)

A technical note aiming to offer deeper intuition for the LayerNorm function common in deep neural networks. LayerNorm is defined relative to a distinguished 'neural' basis, but it does more than just normalize the corresponding vector elements. Rather, it implements a composition -- of linear projection, nonlinear scaling, and then affine transformation -- on input activation vectors. We develop both a new mathematical expression and geometric intuition, to make the net effect more transparent. We emphasize that, when LayerNorm acts on an N-dimensional vector space, all outcomes of LayerNorm lie within the intersection of an (N-1)-dimensional hyperplane and the interior of an N-dimensional hyperellipsoid. This intersection is the interior of an (N-1)-dimensional hyperellipsoid, and typical inputs are mapped near its surface. We find the direction and length of the principal axes of this (N-1)-dimensional hyperellipsoid via the eigen-decomposition of a simply constructed matrix.