mlsanityData to 7 Oct 2026

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

arXiv 1602.07868 ↗First posted 25 Feb 2016cs.LG cs.AI cs.NEcited by 541

Not on the map. Why?

Abstract (arXiv)

We present weight normalization: a reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction. By reparameterizing the weights in this way we improve the conditioning of the optimization problem and we speed up convergence of stochastic gradient descent. Our reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that our method can also be applied successfully to recurrent models such as LSTMs and to noise-sensitive applications such as deep reinforcement learning or generative models, for which batch normalization is less well suited. Although our method is much simpler, it still provides much of the speed-up of full batch normalization. In addition, the computational overhead of our method is lower, permitting more optimization steps to be taken in the same amount of time. We demonstrate the usefulness of our method on applications in supervised image recognition, generative modelling, and deep reinforcement learning.

Key facts. Cited by 541 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Feb 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 126; 2022: 110; 2023: 97; 2024: 86; 2025: 57; 2026: 63.

541 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2022A ConvNet for the 2020scited by 3,545
  5. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  6. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  8. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  9. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  10. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  11. 2022High Fidelity Neural Audio Compressioncited by 673
  12. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
38 more
  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  2. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  3. 2021Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapescited by 426
  4. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  6. 2022Contrastive Test-Time Adaptationcited by 258
  7. 2021Large Scale Image Completion via Co-Modulated Generative Adversarial Networkscited by 219
  8. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  9. 2021Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptationcited by 176
  10. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  11. 2021SMPLicit: Topology-aware Generative Model for Clothed Peoplecited by 154
  12. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  13. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  14. 2022ARAH: Animatable Volume Rendering of Articulated Human SDFscited by 139
  15. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  16. 2023Relightable Gaussian Codec Avatarscited by 126
  17. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  18. 2025Test-Time Training Done Rightcited by 121
  19. 2023An Expert's Guide to Training Physics-informed Neural Networkscited by 119
  20. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  21. 2021Large-Scale Differentially Private BERTcited by 117
  22. 2022Long-Tailed Recognition via Weight Balancingcited by 107
  23. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  24. 2021PhyCRNet: Physics-informed Convolutional-Recurrent Network for Solving Spatiotemporal PDEscited by 97
  25. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  26. 20223D UX-Net: A Large Kernel Volumetric ConvNet Modernizing Hierarchical Transformer for Medical Image Segmentationcited by 94
  27. 2022Attracting and Dispersing: A Simple Approach for Source-free Domain Adaptationcited by 93
  28. 2021NPMs: Neural Parametric Models for 3D Deformable Shapescited by 89
  29. 2021LAPAR: Linearly-Assembled Pixel-Adaptive Regression Network for Single Image Super-Resolution and Beyondcited by 88
  30. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  31. 2022Learning Smooth Neural Functions via Lipschitz Regularizationcited by 86
  32. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  33. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 86
  34. 2021Improved architectures and training algorithms for deep operator networkscited by 85
  35. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  36. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  37. 2021PD-GAN: Probabilistic Diverse GAN for Image Inpaintingcited by 80
  38. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 78

50 most cited of 541 All 541 (JSON)