mlsanityData to 14 Oct 2026

Transformers without Normalization

arXiv 2503.10622 ↗First posted 13 Mar 2025cs.LG cs.AI cs.CLcited by 70

Not on the map. Why?

Abstract (arXiv)

Normalization layers are ubiquitous in modern neural networks and have long been considered essential. This work demonstrates that Transformers without normalization can achieve the same or better performance using a remarkably simple technique. We introduce Dynamic Tanh (DyT), an element-wise operation DyT(x) = tanh(α x), as a drop-in replacement for normalization layers in Transformers. DyT is inspired by the observation that layer normalization in Transformers often produces tanh-like, S-shaped input-output mappings. By incorporating DyT, Transformers without normalization can match or exceed the performance of their normalized counterparts, mostly without hyperparameter tuning. We validate the effectiveness of Transformers with DyT across diverse settings, ranging from recognition to generation, supervised to self-supervised learning, and computer vision to language models. These findings challenge the conventional understanding that normalization layers are indispensable in modern neural networks, and offer new insights into their role in deep networks.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Builds on 66 of its 88 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

66 of 88 references.

References · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  12. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
38 more
  1. 2022Scalable Diffusion Models with Transformerscited by 6,054
  2. 2016Layer Normalizationcited by 5,669
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2022A ConvNet for the 2020scited by 3,579
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023Mistral 7Bcited by 3,164
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  12. 2024Qwen2 Technical Reportcited by 2,120
  13. 2018Group Normalizationcited by 1,524
  14. 2019Root Mean Square Layer Normalizationcited by 1,480
  15. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  16. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  17. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  18. 2016Deep Networks with Stochastic Depthcited by 979
  19. 2020On Layer Normalization in the Transformer Architecturecited by 922
  20. 2017Self-Normalizing Neural Networkscited by 598
  21. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  22. 2024InternLM2 Technical Reportcited by 397
  23. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  24. 2018How Does Batch Normalization Help Optimization?cited by 357
  25. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  26. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  27. 2019Understanding and Improving Layer Normalizationcited by 217
  28. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  29. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  30. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 194
  31. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  32. 2018Understanding Batch Normalizationcited by 159
  33. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 157
  34. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  35. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 113
  36. 2020Normalization Techniques in Training DNNs: Methodology, Analysis and Applicationcited by 90
  37. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 88
  38. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 41; 2026: 29.

70 papers.

Sort:

By area:

  1. 2025ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Codecited by 36
  2. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 23
  3. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Trainingcited by 23
  4. 2025NTIRE 2025 Challenge on Day and Night Raindrop Removal for Dual-Focused Images: Methods and Resultscited by 22
  5. 2025Accurate Parameter-Efficient Test-Time Adaptation for Time Series Forecastingcited by 14
  6. 2025Stronger Normalization-Free Transformerscited by 13
  7. 2025Towards Fully FP8 GEMM LLM Training at Scalecited by 11
  8. 2026SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Normcited by 10
  9. 2025Subspace-Boosted Model Mergingcited by 9
  10. 2025FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillationcited by 9
  11. 2025Learning in Compact Spaces with Approximately Normalized Transformercited by 6
  12. 2025Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformerscited by 6
38 more
  1. 2025SeeDNorm: Self-Rescaled Dynamic Normalizationcited by 6
  2. 2025Weight-Space Linear Recurrent Neural Networkscited by 5
  3. 2026Learning Relativistic Geodesics and Chaotic Dynamics via Stabilized Lagrangian Neural Networkscited by 5
  4. 2025On Surjectivity of Neural Networks: Can you elicit any behavior from your model?cited by 4
  5. 2026Gated Normalization Removal and Scale Anchoring in Pre-Norm Transformerscited by 4
  6. 2026Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Modelscited by 4
  7. 2025Pangu Light: Weight Re-Initialization for Pruning and Accelerating LLMscited by 3
  8. 2025Mamba Goes HoME: Hierarchical Soft Mixture-of-Experts for 3D Medical Image Segmentationcited by 3
  9. 2025Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classificationcited by 3
  10. 2025Spatially Aware Linear Transformer (SAL-T) for Particle Jet Taggingcited by 3
  11. 2026PHAT: Modeling Period Heterogeneity for Multivariate Time Series Forecastingcited by 3
  12. 2026SAME: A Semantically-Aligned Music Autoencodercited by 3
  13. 2025DyTTP: Trajectory Prediction with Normalization-Free Transformerscited by 2
  14. 2025Just One Layer Norm Guarantees Stable Extrapolationcited by 2
  15. 2025Guarded Query Routing for Large Language Modelscited by 2
  16. 2025Taming LLMs by Scaling Learning Rates with Gradient Groupingcited by 2
  17. 2025A Mathematical Explanation of Transformerscited by 2
  18. 2025Bounded Hyperbolic Tangent: A Stable and Efficient Alternative to Pre-Layer Normalization in Large Language Modelscited by 2
  19. 2026Evaluating and Correcting Human Annotation Bias in Dynamic Micro-Expression Recognitioncited by 2
  20. 2026SurGe: Improved Surface Geometry in Point Mapscited by 2
  21. 2025Model Hemorrhage and the Robustness Limits of Large Language Modelscited by 1
  22. 2025Temporal and Spatial Feature Fusion Framework for Dynamic Micro Expression Recognitioncited by 1
  23. 2025MGDFIS: Multi-scale Global-detail Feature Integration Strategy for Small Object Detectioncited by 1
  24. 2025DNT: a Deeply Normalized Transformer that can be trained by Momentum SGDcited by 1
  25. 2025E-CaTCH: Event-Centric Cross-Modal Attention with Temporal Consistency and Class-Imbalance Handling for Misinformation Detectioncited by 1
  26. 2025Every Subtlety Counts: Fine-grained Person Independence Micro-Action Recognition via Distributionally Robust Optimizationcited by 1
  27. 2026Does Your Optimizer Care How You Normalize? Normalization-Optimizer Coupling in LLM Trainingcited by 1
  28. 2025Towards Understanding the Optimization Mechanisms in Deep Learningcited by 0
  29. 2025Expressive yet Efficient Feature Expansion with Adaptive Cross-Hadamard Productscited by 0
  30. 2025Trajectory First: A Curriculum for Discovering Diverse Policiescited by 0
  31. 2025Machine Intelligence on Wireless Edge Networkscited by 0
  32. 2025Multi-Level Fusion Graph Neural Network for Molecule Property Predictioncited by 0
  33. 2025Fractal Flow: Hierarchical and Interpretable Normalizing Flow via Topic Modeling and Recursive Strategycited by 0
  34. 2025DyGLNet: Hybrid Global-Local Feature Fusion with Dynamic Upsampling for Medical Image Segmentationcited by 0
  35. 2025EfficienT-HDR: An Efficient Transformer-Based Framework via Multi-Exposure Fusion for HDR Reconstructioncited by 0
  36. 2025Graph Neural Networks with Diversity-aware Neighbor Selection and Dynamic Multi-scale Fusion for Multivariate Time Series Forecastingcited by 0
  37. 2025GateMABSA: Aspect-Image Gated Fusion for Multimodal Aspect-based Sentiment Analysiscited by 0
  38. 2025Chain-of-Influence: Tracing Interdependencies Across Time and Features in Clinical Predictive Modelingscited by 0

50 most cited of 70 All 70 (JSON)