# Peri-LN: Revisiting Normalization Layer in the Transformer Architecture

arXiv: https://arxiv.org/abs/2502.02732 · HTML: https://mlsanity.com/paper/2502.02732/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 53 references that are in the map.
- First posted on arXiv 4 Feb 2025.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [Massive Activations in Large Language Models](https://mlsanity.com/paper/2402.17762/) (2024)
- [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022)
- [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023)
- [Stabilizing Transformer Training by Preventing Attention Entropy Collapse](https://mlsanity.com/paper/2303.06296/) (2023)
- [Why Transformers Need Adam: A Hessian Perspective](https://mlsanity.com/paper/2402.16788/) (2024)
- [Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization](https://mlsanity.com/paper/2305.14152/) (2023)
- [Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks](https://mlsanity.com/paper/2310.02244/) (2023)
- [nGPT: Normalized Transformer with Representation Learning on the Hypersphere](https://mlsanity.com/paper/2410.01131/) (2024)
- [Scaling FP8 training to trillion-token LLMs](https://mlsanity.com/paper/2409.12517/) (2024)

## Built on by (most cited first, 30 of 34)

- [Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers](https://mlsanity.com/paper/2606.18206/) (2026)
- [Arcee Trinity Large Technical Report](https://mlsanity.com/paper/2602.17004/) (2026)
- [Learning to Skip the Middle Layers of Transformers](https://mlsanity.com/paper/2506.21103/) (2025)
- [Hidden Dynamics of Massive Activations in Transformer Training](https://mlsanity.com/paper/2508.03616/) (2025)
- [SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm](https://mlsanity.com/paper/2602.08064/) (2026)
- [Stability and Generalization in Looped Transformers](https://mlsanity.com/paper/2604.15259/) (2026)
- [Optimal Scaling Needs Optimal Norm](https://mlsanity.com/paper/2510.03871/) (2025)
- [The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss](https://mlsanity.com/paper/2512.08374/) (2025)
- [Semantic Routing: Exploring Multi-Layer LLM Feature Weighting for Diffusion Transformers](https://mlsanity.com/paper/2602.03510/) (2026)
- [Gated Normalization Removal and Scale Anchoring in Pre-Norm Transformers](https://mlsanity.com/paper/2602.10408/) (2026)
- [Controlling changes to attention logits](https://mlsanity.com/paper/2511.21377/) (2025)
- [Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale](https://mlsanity.com/paper/2606.19491/) (2026)
- [Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors](https://mlsanity.com/paper/2606.25971/) (2026)
- [GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling](https://mlsanity.com/paper/2506.22049/) (2025)
- [GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Models](https://mlsanity.com/paper/2512.15560/) (2025)
- [Bounded Hyperbolic Tangent: A Stable and Efficient Alternative to Pre-Layer Normalization in Large Language Models](https://mlsanity.com/paper/2601.09719/) (2025)
- [SpanNorm: Reconciling Training Stability and Performance in Deep Transformers](https://mlsanity.com/paper/2601.22580/) (2026)
- [Explaining Grokking in Transformers through the Lens of Inductive Bias](https://mlsanity.com/paper/2602.06702/) (2026)
- [Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth](https://mlsanity.com/paper/2607.14018/) (2026)
- [GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization](https://mlsanity.com/paper/2601.22095/) (2026)
- [UDAPose: Unsupervised Domain Adaptation for Low-Light Human Pose Estimation](https://mlsanity.com/paper/2604.10485/) (2026)
- [Dead Directions: Geometric Singular Learning](https://mlsanity.com/paper/2606.05957/) (2026)
- [DepthBench: Measuring How Residual Connections Enable More Computational Depth](https://mlsanity.com/paper/2609.32534/) (2026)
- [Post-Norm can Resharpen Attention](https://mlsanity.com/paper/2510.08341/) (2025)
- [Stabilizing Transformer Training Through Consensus](https://mlsanity.com/paper/2601.22614/) (2026)
- [When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizer](https://mlsanity.com/paper/2604.23434/) (2026)
- [FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs](https://mlsanity.com/paper/2606.19025/) (2026)
- [Neural Texture Compression using Hypernetworks](https://mlsanity.com/paper/2606.26913/) (2026)
- [Sharp Stability Threshold and Certification for Designing Stable Residual Architectures](https://mlsanity.com/paper/2607.14576/) (2026)
- [Sphere Retraction Normalizations](https://mlsanity.com/paper/2608.02668/) (2026)

## Abstract (arXiv)

Selecting a layer normalization (LN) strategy that stabilizes training and speeds convergence in Transformers remains difficult, even for today's large language models (LLM). We present a comprehensive analytical foundation for understanding how different LN strategies influence training dynamics in large-scale Transformers. Until recently, Pre-LN and Post-LN have long dominated practices despite their limitations in large-scale training. However, several open-source models have recently begun silently adopting a third strategy without much explanation. This strategy places normalization layer peripherally around sublayers, a design we term Peri-LN. While Peri-LN has demonstrated promising performance, its precise mechanisms and benefits remain almost unexplored. Our in-depth analysis delineates the distinct behaviors of LN strategies, showing how each placement shapes activation variance and gradient propagation. To validate our theoretical insight, we conduct extensive experiments on Transformers up to 3.2B parameters, showing that Peri-LN consistently achieves more balanced variance growth, steadier gradient flow, and convergence stability. Our results suggest that Peri-LN warrants broader consideration for large-scale Transformer architectures, providing renewed insights into the optimal placement of LN.