mlsanityData to 7 Oct 2026

Root Mean Square Layer Normalization

arXiv 1910.07467 ↗First posted 16 Oct 2019cs.LG cs.CL stat.MLcited by 1,425

Not on the map. Why?

Cited by 121 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Layer normalization (LayerNorm) has been successfully applied to various deep neural networks to help stabilize training and boost model convergence because of its capability in handling re-centering and re-scaling of both inputs and weight matrix. However, the computational overhead introduced by LayerNorm makes these improvements expensive and significantly slows the underlying network, e.g. RNN in particular. In this paper, we hypothesize that re-centering invariance in LayerNorm is dispensable and propose root mean square layer normalization, or RMSNorm. RMSNorm regularizes the summed inputs to a neuron in one layer according to root mean square (RMS), giving the model re-scaling invariance property and implicit learning rate adaptation ability. RMSNorm is computationally simpler and thus more efficient than LayerNorm. We also present partial RMSNorm, or pRMSNorm where the RMS is estimated from p% of the summed inputs without breaking the above properties. Extensive experiments on several tasks using diverse network architectures show that RMSNorm achieves comparable performance against LayerNorm but reduces the running time by 7%~64% on different models. Source code is available at https://github.com/bzhangGo/rmsnorm.

Key facts. Cited by 1,425 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 11; 2023: 84; 2024: 302; 2025: 463; 2026: 557.

1,425 papers.

Sort:

By area:

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2025Gemma 3 Technical Reportcited by 1,686
  7. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  8. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2025Qwen-Image Technical Reportcited by 973
  11. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
38 more
  1. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  2. 2025Large Language Diffusion Modelscited by 872
  3. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  4. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  5. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  6. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  7. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  8. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  9. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  10. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  11. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  12. 2023A Comprehensive Overview of Large Language Modelscited by 639
  13. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  14. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  15. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  16. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
  17. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  18. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  19. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  20. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  21. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  22. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  23. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  24. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 376
  25. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  26. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  27. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  28. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  29. 2024Titans: Learning to Memorize at Test Timecited by 303
  30. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  31. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  32. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  33. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  34. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 280
  35. 2024Massive Activations in Large Language Modelscited by 277
  36. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  37. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  38. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242

50 most cited of 1,425 All 1,425 (JSON)