# Weight Norm Control

arXiv: https://arxiv.org/abs/2311.11446 · HTML: https://mlsanity.com/paper/2311.11446/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 10 references that are in the map.
- First posted on arXiv 19 Nov 2023.

## Builds on (most cited first, 3 of 3)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [Improving Deep Learning Optimization through Constrained Parameter Regularization](https://mlsanity.com/paper/2311.09058/) (2023)

## Built on by (most cited first, 4 of 4)

- [nGPT: Normalized Transformer with Representation Learning on the Hypersphere](https://mlsanity.com/paper/2410.01131/) (2024)
- [Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining](https://mlsanity.com/paper/2608.24814/) (2026)
- [Decoupled Orthogonal Dynamics: Regularization for Deep Network Optimizers](https://mlsanity.com/paper/2602.05136/) (2026)
- [Training nGPT](https://mlsanity.com/paper/2608.01284/) (2026)

## Abstract (arXiv)

We note that decoupled weight decay regularization is a particular case of weight norm control where the target norm of weights is set to 0. Any optimization method (e.g., Adam) which uses decoupled weight decay regularization (respectively, AdamW) can be viewed as a particular case of a more general algorithm with weight norm control (respectively, AdamWN). We argue that setting the target norm of weights to 0 can be suboptimal and other target norm values can be considered. For instance, any training run where AdamW achieves a particular norm of weights can be challenged by AdamWN scheduled to achieve a comparable norm of weights. We discuss various implications of introducing weight norm control instead of weight decay.