mlsanityData to 7 Oct 2026

Equi-normalization of Neural Networks

arXiv 1902.10416 ↗First posted 27 Feb 2019cs.CV cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Modern neural networks are over-parametrized. In particular, each rectified linear hidden unit can be modified by a multiplicative factor by adjusting input and output weights, without changing the rest of the network. Inspired by the Sinkhorn-Knopp algorithm, we introduce a fast iterative method for minimizing the L2 norm of the weights, equivalently the weight decay regularizer. It provably converges to a unique solution. Interleaving our algorithm with SGD during training improves the test accuracy. For small batches, our approach offers an alternative to batch-and group-normalization on CIFAR-10 and ImageNet with a ResNet-18.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 2; 2024: 2; 2025: 4; 2026: 5.

16 papers.

Sort:

  1. 2025Symmetry in Neural Network Parameter Spacescited by 31
  2. 2024Scale Equivariant Graph Metanetworkscited by 23
  3. 2021An Embedding of ReLU Networks and an Analysis of their Identifiabilitycited by 20
  4. 2025Transformative or Conservative? Conservation laws for ResNets and Transformerscited by 7
  5. 2022REx: Data-Free Residual Quantization Error Expansioncited by 6
  6. 2026Path-conditioned training: a principled way to rescale ReLU neural networkscited by 6
  7. 2023Exploring the Complexity of Deep Neural Networks through Functional Equivalencecited by 4
  8. 2022PathProx: A Proximal Gradient Algorithm for Weight Decay Regularized Deep Neural Networkscited by 2
  9. 2024Efficient Algorithms for Regularized Nonnegative Scale-invariant Low-rank Approximation Modelscited by 2
  10. 2025Improving Learning to Optimize Using Parameter Symmetriescited by 2
  11. 2025Non-Vacuous Generalization Bounds: Can Rescaling Invariances Help?cited by 1
  12. 2026Conservation Laws for Modern Neural Architecturescited by 1
4 more
  1. 2023PIPE : Parallelized Inference Through Post-Training Quantization Ensembling of Residual Expansionscited by 0
  2. 2026Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuningcited by 0
  3. 2026Dense Structural Compression of Transformers via Gauge-Correct Channel Removalcited by 0
  4. 2026Stability-Constrained Approximation in Spline KANs: Exact Layer Balancing and Budget-Compatible Saturationcited by 0