mlsanityData to 7 Oct 2026

GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization

arXiv 2601.22095 ↗First posted 29 Jan 2026cs.LG cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

The placement of normalization layers, specifically Pre-Norm and Post-Norm, remains an open question in Transformer architecture design. In this work, we rethink these approaches through the lens of manifold optimization, interpreting the outputs of the Feed-Forward Network (FFN) and attention layers as update directions in optimization. Building on this perspective, we introduce GeoNorm, a novel method that replaces standard normalization with geodesic updates on the manifold. Furthermore, analogous to learning rate schedules, we propose a layer-wise update decay for the FFN and attention components. Comprehensive experiments demonstrate that GeoNorm consistently outperforms existing normalization methods in Transformer models. Crucially, GeoNorm can be seamlessly integrated into standard Transformer architectures, achieving performance improvements with negligible additional computational cost.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 75 of its 91 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

75 of 91 references.

References · 75

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2025Qwen3 Technical Reportcited by 8,195
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2022A ConvNet for the 2020scited by 3,545
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
38 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2024Mixtral of Expertscited by 1,785
  8. 2018Group Normalizationcited by 1,514
  9. 2019Root Mean Square Layer Normalizationcited by 1,425
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  13. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  14. 2020On Layer Normalization in the Transformer Architecturecited by 906
  15. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  16. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  17. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  18. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  19. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  20. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  21. 2025Kimi K2: Open Agentic Intelligencecited by 523
  22. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  23. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  24. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  25. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  26. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  27. 2020Query-Key Normalization for Transformerscited by 266
  28. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  29. 2023From Sparse to Soft Mixtures of Expertscited by 245
  30. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  31. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  32. 2020Understanding the Difficulty of Training Transformerscited by 210
  33. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  34. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  35. 2018Understanding Batch Normalizationcited by 157
  36. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  37. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  38. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Rethinking Normalization Placement for LLMs: Post-Norm under Curriculum Depth Growingcited by 0