mlsanityData to 7 Oct 2026

Fixup Initialization: Residual Learning Without Normalization

arXiv 1901.09321 ↗First posted 27 Jan 2019cs.LG cs.CV stat.MLcited by 194

Not on the map. Why?

Abstract (arXiv)

Normalization layers are a staple in state-of-the-art deep neural network architectures. They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic. In this work, we challenge the commonly-held beliefs by showing that none of the perceived benefits is unique to normalization. Specifically, we propose fixed-update initialization (Fixup), an initialization motivated by solving the exploding and vanishing gradient problem at the beginning of training via properly rescaling a standard initialization. We find training residual networks with Fixup to be as stable as training with normalization -- even for networks with 10,000 layers. Furthermore, with proper regularization, Fixup enables residual networks without normalization to achieve state-of-the-art performance in image classification and machine translation.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 36; 2022: 36; 2023: 47; 2024: 11; 2025: 23; 2026: 40.

194 papers.

Sort:

By area:

  1. 2021Going deeper with Image Transformerscited by 759
  2. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  3. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  4. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 216
  5. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  6. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  7. 2023Efficient Large Language Models: A Surveycited by 183
  8. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  9. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  10. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  11. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  12. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
38 more
  1. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  2. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  3. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  4. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  5. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  6. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  7. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  8. 2025Transformers without Normalizationcited by 69
  9. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  10. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  11. 2022Do Bayesian Neural Networks Need To Be Fully Stochastic?cited by 63
  12. 2022Fast Finite Width Neural Tangent Kernelcited by 62
  13. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 56
  14. 2021A Survey on Green Deep Learningcited by 52
  15. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  16. 2021Free-viewpoint Indoor Neural Relighting from Multi-view Stereocited by 50
  17. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  18. 2022Understanding AdamW through Proximal Methods and Scale-Freenesscited by 46
  19. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
  20. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
  21. 2022Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximationscited by 42
  22. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  23. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 39
  24. 2023Scaling laws for single-agent reinforcement learningcited by 39
  25. 2021Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equationscited by 38
  26. 2023A Survey on Efficient Training of Transformerscited by 37
  27. 2023Simplifying Transformer Blockscited by 37
  28. 2021Fast Axiomatic Attribution for Neural Networkscited by 35
  29. 2026Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformerscited by 34
  30. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  31. 2021KNAS: Green Neural Architecture Searchcited by 33
  32. 2023LipsFormer: Introducing Lipschitz Continuity to Vision Transformerscited by 33
  33. 2021StarEnhancer: Learning Real-Time and Style-Aware Image Enhancementcited by 32
  34. 2023EverLight: Indoor-Outdoor Editable HDR Lighting Estimationcited by 32
  35. 2022Acceleration of Federated Learning with Alleviated Forgetting in Local Trainingcited by 29
  36. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  37. 2023Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?cited by 29
  38. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29

50 most cited of 194 All 194 (JSON)