mlsanityData to 14 Oct 2026

DeepNet: Scaling Transformers to 1,000 Layers

arXiv 2203.00555 ↗First posted 1 Mar 2022cs.CL cs.LGcited by 178

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction.

Key facts. Cited by 178 papers in the map (data through 14 Oct 2026). Builds on 46 of its 74 references that are in the map. First posted on arXiv 1 Mar 2022.

Builds on

46 of 74 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
34 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  7. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  8. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  9. 2020On Layer Normalization in the Transformer Architecturecited by 922
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  11. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  12. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  13. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  14. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  15. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  16. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  17. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  18. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  19. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  20. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  21. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  22. 2020Understanding the Difficulty of Training Transformerscited by 210
  23. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  24. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  25. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  26. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  27. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  28. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 126
  29. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  30. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  31. 2019Visualizing and Understanding the Effectiveness of BERTcited by 67
  32. 2019Improving Deep Transformer with Depth-Scaled Initialization and Merged Attentioncited by 59
  33. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  34. 2020Optimizing Deeper Transformers on Small Datasetscited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 32; 2023: 39; 2024: 17; 2025: 26; 2026: 64.

178 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  5. 2023A Comprehensive Overview of Large Language Modelscited by 648
  6. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  7. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  8. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  9. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  10. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  11. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  12. 2022Pure Transformers are Powerful Graph Learnerscited by 188
38 more
  1. 2023Efficient Large Language Models: A Surveycited by 186
  2. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 181
  3. 2025Hierarchical Reasoning Modelcited by 177
  4. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 161
  5. 2023Spike-driven Transformercited by 146
  6. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  7. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 124
  8. 2022Language Models are General-Purpose Interfacescited by 97
  9. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  10. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 78
  11. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  12. 2026Attention Residualscited by 65
  13. 2025The Curse of Depth in Large Language Modelscited by 62
  14. 2025Large language models for automated scholarly paper review: A surveycited by 61
  15. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 60
  16. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  17. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 51
  18. 2024xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Proteincited by 50
  19. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49
  20. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  21. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 45
  22. 2022VL-BEiT: Generative Vision-Language Pretrainingcited by 39
  23. 2023Simplifying Transformer Blockscited by 37
  24. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
  25. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  26. 2022SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translationscited by 35
  27. 2023LipsFormer: Introducing Lipschitz Continuity to Vision Transformerscited by 34
  28. 2023ResiDual: Transformer with Dual Residual Connectionscited by 31
  29. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 28
  30. 2022Insights into Pre-training via Simpler Synthetic Taskscited by 26
  31. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  32. 2023VanillaNet: the Power of Minimalism in Deep Learningcited by 23
  33. 2023Implicit regularization of deep residual networks towards neural ODEscited by 20
  34. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 20
  35. 2022Foundation Transformerscited by 19
  36. 2022Scaling ResNets in the Large-depth Regimecited by 18
  37. 2022Learning from partially labeled data for multi-organ and tumor segmentationcited by 18
  38. 2025ViT3: Unlocking Test-Time Training in Visioncited by 18

50 most cited of 178 All 178 (JSON)