mlsanityData to 7 Oct 2026

ReZero is All You Need: Fast Convergence at Large Depth

arXiv 2003.04887 ↗First posted 10 Mar 2020cs.LG cs.CL stat.MLcited by 260

Not on the map. Why?

Abstract (arXiv)

Deep networks often suffer from vanishing or exploding gradients due to inefficient signal propagation, leading to long training times or convergence difficulties. Various architecture designs, sophisticated residual-style networks, and initialization schemes have been shown to improve deep signal propagation. Recently, Pennington et al. used free probability theory to show that dynamical isometry plays an integral role in efficient deep learning. We show that the simplest architecture change of gating each residual connection using a single zero-initialized parameter satisfies initial dynamical isometry and outperforms more complex approaches. Although much simpler than its predecessors, this gate enables training thousands of fully connected layers with fast convergence and better test performance for ResNets trained on CIFAR-10. We apply this technique to language modeling and find that we can easily train 120-layer Transformers. When applied to 12 layer Transformers, it converges 56% faster on enwiki8.

Key facts. Cited by 260 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 35; 2022: 35; 2023: 47; 2024: 44; 2025: 26; 2026: 73.

260 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2021Going deeper with Image Transformerscited by 759
  3. 2021A Survey of Transformerscited by 399
  4. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  5. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  6. 2021Generating Images with Sparse Representationscited by 262
  7. 2023Efficient Large Language Models: A Surveycited by 183
  8. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  9. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  10. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  11. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  12. 2021MoViNets: Mobile Video Networks for Efficient Video Recognitioncited by 135
38 more
  1. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  2. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  3. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 106
  4. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  5. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  6. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  7. 2022Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encodingcited by 83
  8. 2023BQ-NCO: Bisimulation Quotienting for Efficient Neural Combinatorial Optimizationcited by 80
  9. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  10. 2023GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learningcited by 70
  11. 2025Transformers without Normalizationcited by 69
  12. 2023Towards Omni-generalizable Neural Methods for Vehicle Routing Problemscited by 68
  13. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
  14. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  15. 2021Self-Supervised Bug Detection and Repaircited by 59
  16. 2026Attention Residualscited by 57
  17. 2023Context-I2W: Mapping Images to Context-dependent Words for Accurate Zero-Shot Composed Image Retrievalcited by 52
  18. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
  19. 2024MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereocited by 49
  20. 2022Transformers in Time-series Analysis: A Tutorialcited by 45
  21. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
  22. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
  23. 2022UVCGAN: UNet Vision Transformer cycle-consistent GAN for unpaired image-to-image translationcited by 42
  24. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  25. 2024GOAL: A Generalist Combinatorial Optimization Agent Learnercited by 39
  26. 2023A Survey on Efficient Training of Transformerscited by 37
  27. 2023Simplifying Transformer Blockscited by 37
  28. 2021Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphscited by 36
  29. 2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 36
  30. 2025Rethinking Light Decoder-based Solvers for Vehicle Routing Problemscited by 36
  31. 2022On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Modelscited by 34
  32. 2023ExcelFormer: A neural network surpassing GBDTs on tabular datacited by 33
  33. 2023LipsFormer: Introducing Lipschitz Continuity to Vision Transformerscited by 33
  34. 2024Self-Improvement for Neural Combinatorial Optimization: Sample without Replacement, but Improvementcited by 32
  35. 2023ResiDual: Transformer with Dual Residual Connectionscited by 30
  36. 2021Latent Space Data Assimilation by using Deep Learningcited by 29
  37. 2021ZerO Initialization: Initializing Neural Networks with only Zeros and Onescited by 28
  38. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28

50 most cited of 260 All 260 (JSON)