mlsanityData to 7 Oct 2026

BranchNorm: Robustly Scaling Extremely Deep Transformers

arXiv 2305.02790 ↗First posted 4 May 2023cs.LG cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Recently, DeepNorm scales Transformers into extremely deep (i.e., 1000 layers) and reveals the promising potential of deep scaling. To stabilize the training of deep models, DeepNorm (Wang et al., 2022) attempts to constrain the model update to a constant value. Although applying such a constraint can benefit the early stage of model training, it may lead to undertrained models during the whole training procedure. In this paper, we propose BranchNorm, which dynamically rescales the non-residual branch of Transformer in accordance with the training period. BranchNorm not only theoretically stabilizes the training with smooth gradient norms at the early stage, but also encourages better convergence in the subsequent training stage. Experiment results on multiple translation tasks demonstrate that BranchNorm achieves a better trade-off between training stability and converge performance.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

30 of 42 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2016Layer Normalizationcited by 5,579
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2020Rethinking Attention with Performerscited by 1,584
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  10. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  11. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  12. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
18 more
  1. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  2. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  3. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  4. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  5. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  6. 2020Understanding the Difficulty of Training Transformerscited by 210
  7. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  8. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  9. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  10. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  11. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  12. 2019Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlationscited by 61
  13. 2019Improving Deep Transformer with Depth-Scaled Initialization and Merged Attentioncited by 59
  14. 2018MTNT: A Testbed for Machine Translation of Noisy Textcited by 54
  15. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
  16. 2021M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretrainingcited by 31
  17. 2021Mask Attention Networks: Rethinking and Strengthen Transformercited by 28
  18. 2021Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variablescited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 2.

6 papers.

  1. 2024Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learningcited by 4
  2. 2025Spectral Alignment as Predictor of Loss Explosion in Neural Network Trainingcited by 2
  3. 2026Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depthcited by 2
  4. 2024LCS: A Language Converter Strategy for Zero-Shot Neural Machine Translationcited by 1
  5. 2025Model Hemorrhage and the Robustness Limits of Large Language Modelscited by 1
  6. 2026Progressive Residual Warmup for Language Model Pretrainingcited by 1