Rethinking Normalization Placement for LLMs: Post-Norm under Curriculum Depth Growing
Not on the map. Why?
Abstract (arXiv)
Pre-norm is the standard normalization placement in modern Transformers because it facilitates joint optimization of full-depth models. We ask whether this preference persists when depth is introduced through a curriculum. In curriculum depth growth, each appended block receives the boundary representation produced by a trained prefix, making normalization placement relevant to forward conditioning. We therefore test whether placement and training curriculum interact. In a controlled distillation study with a Qwen3-8B teacher and a nine-layer student, pre-norm and post-norm are indistinguishable under joint training, differing by 0.0004 validation CE, while post-norm improves over pre-norm by 0.0328 under curriculum growth, an order of magnitude larger. A post-joint control matched by student active-layer tokens remains worse than post-grow, which rules out compute as the sole explanation. The ranking crosses over during the curriculum: post-norm takes the lead once blocks are appended. Single-block and freeze controls localize the ranking change to block appending rather than shallow-block quality or retraining. Boundary diagnostics associate post-norm with stable residual scales and pre-norm with structural-token scale drift; on a fixed batch, the final pre-grow block is also nearly identity-mapped. Together with the phase-wise crossover, these observations are consistent with boundary-scale conditioning after new blocks are appended. The results motivate treating normalization placement and training curriculum as coupled design choices in this distillation setting.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 13 Aug 2026.
Builds on
21 of 27 references.
References · 21
- 2017Attention Is All You Needcited by 60,361
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
- 2020GLU Variants Improve Transformercited by 1,804
- 2019Root Mean Square Layer Normalizationcited by 1,480
- 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
- 2020On Layer Normalization in the Transformer Architecturecited by 922
- 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
- 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
- 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
9 more
- 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
- 2020Understanding the Difficulty of Training Transformerscited by 210
- 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
- 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
- 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
- 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 70
- 2025Peri-LN: Revisiting Normalization Layer in the Transformer Architecturecited by 34
- 2025Stability of Transformers under Layer Normalizationcited by 5
- 2026GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimizationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Gated Normalization Removal and Scale Anchoring in Pre-Norm Transformers
- 2025Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities
- 2026SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm
- 2026LayerRoPE: Dynamic Depth-wise Magnitude & Angular Superposition
- 2026What Visual Generators Need from Teachers: Rethinking Representation Alignment
- 2026Pretraining Transformers with Quantized Softmax in Attention
- 2026Similar Models Learn Differently: Final-Window Pretraining Shapes Post-Training Beyond SFT
- 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes