mlsanityData to 14 Oct 2026

Transformers without Tears: Improving the Normalization of Self-Attention

arXiv 1910.05895 ↗First posted 14 Oct 2019cs.CL cs.LG stat.MLcited by 153

Not on the map. Why?

Abstract (arXiv)

We evaluate three simple, normalization-centric changes to improve Transformer training. First, we show that pre-norm residual connections (PreNorm) and smaller initializations enable warmup-free, validation-based training with large learning rates. Second, we propose ℓ2 normalization with a single scale parameter (ScaleNorm) for faster training and better performance. Finally, we reaffirm the effectiveness of normalizing word embeddings to a fixed length (FixNorm). On five low-resource translation pairs from TED Talks-based corpora, these changes always converge, giving an average +1.1 BLEU over state-of-the-art bilingual baselines and a new 32.8 BLEU on IWSLT'15 English-Vietnamese. We observe sharper performance curves, more consistent gradient norms, and a linear relationship between activation scaling and decoder depth. Surprisingly, in the high-resource setting (WMT'14 English-German), ScaleNorm and FixNorm remain competitive but PreNorm degrades performance.

Key facts. Cited by 153 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 29; 2022: 26; 2023: 26; 2024: 24; 2025: 24; 2026: 24.

153 papers.

Sort:

By area:

  1. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  2. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  3. 2023A Comprehensive Overview of Large Language Modelscited by 648
  4. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  5. 2021A Survey of Transformerscited by 401
  6. 2021Choose a Transformer: Fourier or Galerkincited by 343
  7. 2021Remote Sensing Image Change Detection with Transformerscited by 306
  8. 2022Transformer Quality in Linear Timecited by 191
  9. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  10. 2023GenCast: Diffusion-based ensemble forecasting for medium-range weathercited by 177
  11. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  12. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
38 more
  1. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
  2. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 112
  3. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 104
  4. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 88
  5. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 86
  6. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  7. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  8. 2025Transformers without Normalizationcited by 70
  9. 2023Tighter Bounds on the Expressivity of Transformer Encoderscited by 66
  10. 2026Attention Residualscited by 65
  11. 2025The Curse of Depth in Large Language Modelscited by 62
  12. 2021Keyword Transformer: A Self-Attention Model for Keyword Spottingcited by 60
  13. 2022PinnerFormer: Sequence Modeling for User Representation at Pinterestcited by 57
  14. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 52
  15. 2021Long-Range Transformers for Dynamic Spatiotemporal Forecastingcited by 46
  16. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 41
  17. 2025In Search of Adam's Secret Saucecited by 40
  18. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 39
  19. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  20. 2021Fast Nearest Neighbor Machine Translationcited by 37
  21. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
  22. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 36
  23. 2021KNAS: Green Neural Architecture Searchcited by 33
  24. 2021Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERTcited by 31
  25. 2022Pretrained Models for Multilingual Federated Learningcited by 30
  26. 2023Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammarscited by 30
  27. 2021Variational Graph Normalized Auto-Encoderscited by 29
  28. 2021Attention Approximates Sparse Distributed Memorycited by 29
  29. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 28
  30. 2023Using Sequences of Life-events to Predict Human Livescited by 27
  31. 2021LegoFormer: Transformers for Block-by-Block Multi-view 3D Reconstructioncited by 25
  32. 2024Apple Intelligence Foundation Language Modelscited by 24
  33. 2024Training Neural Networks as Recognizers of Formal Languagescited by 24
  34. 2021Pretraining the Noisy Channel Model for Task-Oriented Dialoguecited by 23
  35. 2021Rethinking Skip Connection with Layer Normalization in Transformers and ResNetscited by 22
  36. 2022Transformer Meets Boundary Value Inverse Problemscited by 22
  37. 2023DuETT: Dual Event Time Transformer for Electronic Health Recordscited by 17
  38. 2023Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patternscited by 17

50 most cited of 153 All 153 (JSON)