# Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations

arXiv: https://arxiv.org/abs/2008.02965 · HTML: https://mlsanity.com/paper/2008.02965/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2020.

## Built on by (most cited first, 9 of 9)

- [TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization](https://mlsanity.com/paper/2303.11135/) (2023)
- [The Pitfalls and Promise of Conformal Inference Under Adversarial Attacks](https://mlsanity.com/paper/2405.08886/) (2024)
- [Boosting Adversarial Robustness From The Perspective of Effective Margin Regularization](https://mlsanity.com/paper/2210.05118/) (2022)
- [Variance Control via Weight Rescaling in LLM Pre-training](https://mlsanity.com/paper/2503.17500/) (2025)
- [Clustering Effect of (Linearized) Adversarial Robust Models](https://mlsanity.com/paper/2111.12922/) (2021)
- [PathProx: A Proximal Gradient Algorithm for Weight Decay Regularized Deep Neural Networks](https://mlsanity.com/paper/2210.03069/) (2022)
- [Weight Rescaling: Effective and Robust Regularization for Deep Neural Networks with Batch Normalization](https://mlsanity.com/paper/2102.03497/) (2021)
- [Lipschitz Constant Meets Condition Number: Learning Robust and Compact Deep Neural Networks](https://mlsanity.com/paper/2503.20454/) (2025)
- [Recovering Plasticity of Neural Networks via Soft Weight Rescaling](https://mlsanity.com/paper/2507.04683/) (2025)

## Abstract (arXiv)

Using weight decay to penalize the L2 norms of weights in neural networks has been a standard training practice to regularize the complexity of networks. In this paper, we show that a family of regularizers, including weight decay, is ineffective at penalizing the intrinsic norms of weights for networks with positively homogeneous activation functions, such as linear, ReLU and max-pooling functions. As a result of homogeneity, functions specified by the networks are invariant to the shifting of weight scales between layers. The ineffective regularizers are sensitive to such shifting and thus poorly regularize the model capacity, leading to overfitting. To address this shortcoming, we propose an improved regularizer that is invariant to weight scale shifting and thus effectively constrains the intrinsic norm of a neural network. The derived regularizer is an upper bound for the input gradient of the network so minimizing the improved regularizer also benefits the adversarial robustness. Residual connections are also considered and we show that our regularizer also forms an upper bound to input gradients of such a residual network. We demonstrate the efficacy of our proposed regularizer on various datasets and neural network architectures at improving generalization and adversarial robustness.