# Equi-normalization of Neural Networks

arXiv: https://arxiv.org/abs/1902.10416 · HTML: https://mlsanity.com/paper/1902.10416/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2019.

## Built on by (most cited first, 16 of 16)

- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [Scale Equivariant Graph Metanetworks](https://mlsanity.com/paper/2406.10685/) (2024)
- [An Embedding of ReLU Networks and an Analysis of their Identifiability](https://mlsanity.com/paper/2107.09370/) (2021)
- [Transformative or Conservative? Conservation laws for ResNets and Transformers](https://mlsanity.com/paper/2506.06194/) (2025)
- [REx: Data-Free Residual Quantization Error Expansion](https://mlsanity.com/paper/2203.14645/) (2022)
- [Path-conditioned training: a principled way to rescale ReLU neural networks](https://mlsanity.com/paper/2602.19799/) (2026)
- [Exploring the Complexity of Deep Neural Networks through Functional Equivalence](https://mlsanity.com/paper/2305.11417/) (2023)
- [PathProx: A Proximal Gradient Algorithm for Weight Decay Regularized Deep Neural Networks](https://mlsanity.com/paper/2210.03069/) (2022)
- [Efficient Algorithms for Regularized Nonnegative Scale-invariant Low-rank Approximation Models](https://mlsanity.com/paper/2403.18517/) (2024)
- [Improving Learning to Optimize Using Parameter Symmetries](https://mlsanity.com/paper/2504.15399/) (2025)
- [Non-Vacuous Generalization Bounds: Can Rescaling Invariances Help?](https://mlsanity.com/paper/2509.26149/) (2025)
- [Conservation Laws for Modern Neural Architectures](https://mlsanity.com/paper/2606.17816/) (2026)
- [PIPE : Parallelized Inference Through Post-Training Quantization Ensembling of Residual Expansions](https://mlsanity.com/paper/2311.15806/) (2023)
- [Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning](https://mlsanity.com/paper/2604.22838/) (2026)
- [Dense Structural Compression of Transformers via Gauge-Correct Channel Removal](https://mlsanity.com/paper/2609.07264/) (2026)
- [Stability-Constrained Approximation in Spline KANs: Exact Layer Balancing and Budget-Compatible Saturation](https://mlsanity.com/paper/2609.17619/) (2026)

## Abstract (arXiv)

Modern neural networks are over-parametrized. In particular, each rectified linear hidden unit can be modified by a multiplicative factor by adjusting input and output weights, without changing the rest of the network. Inspired by the Sinkhorn-Knopp algorithm, we introduce a fast iterative method for minimizing the L2 norm of the weights, equivalently the weight decay regularizer. It provably converges to a unique solution. Interleaving our algorithm with SGD during training improves the test accuracy. For small batches, our approach offers an alternative to batch-and group-normalization on CIFAR-10 and ImageNet with a ResNet-18.