# Adaptive Norm-Based Regularization for Neural Networks

arXiv: https://arxiv.org/abs/2605.00171 · HTML: https://mlsanity.com/paper/2605.00171/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 1 of its 5 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Builds on (most cited first, 1 of 1)

- [L2 Regularization versus Batch and Weight Normalization](https://mlsanity.com/paper/1706.05350/) (2017)

## Abstract (arXiv)

In this paper, we study norm-based regularization methods for neural networks. We compare existing penalization approaches and introduce two regularization strategies that extend classical ridge- and lasso-type penalties to neural network models. The first strategy modifies weight decay by incorporating the covariance structure of the input features into a ridge-type ℓ2 penalty, allowing regularization to account for feature dependence. The second combines an ℓ1 sparsity penalty with covariance-aware ℓ2 regularization, producing neural network weights that are both sparse and structurally informed. Monte Carlo simulations are used to evaluate these methods under different data-generating settings, followed by two real-data applications on building cooling-load prediction and leukemia cell-type classification from high-dimensional gene expression data. Across simulated and real-data examples, the proposed regularizers improve predictive performance on unseen data and provide more effective complexity control than standard norm-based penalties, particularly when features are correlated or high-dimensional.