# OUI Need to Talk About Weight Decay: A New Perspective on Overfitting Detection

arXiv: https://arxiv.org/abs/2504.17160 · HTML: https://mlsanity.com/paper/2504.17160/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 24 Apr 2025.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [Why Do We Need Weight Decay in Modern Deep Learning?](https://mlsanity.com/paper/2310.04415/) (2023)
- [On the training dynamics of deep networks with L2 regularization](https://mlsanity.com/paper/2006.08643/) (2020)
- [Theory of Deep Learning III: explaining the non-overfitting puzzle](https://mlsanity.com/paper/1801.00173/) (2017)
- [Adaptive Weight Decay for Deep Neural Networks](https://mlsanity.com/paper/1907.08931/) (2019)
- [Improving Robustness with Adaptive Weight Decay](https://mlsanity.com/paper/2210.00094/) (2022)
- [Quantifying Overfitting: Evaluating Neural Network Performance through Analysis of Null Space](https://mlsanity.com/paper/2305.19424/) (2023)
- [GreenLightningAI: An Efficient AI System with Decoupled Structural and Quantitative Knowledge](https://mlsanity.com/paper/2312.09971/) (2023)

## Built on by (most cited first, 7 of 7)

- [Sinusoidal Initialization, Time for a New Start](https://mlsanity.com/paper/2505.12909/) (2025)
- [Regime Change Hypothesis: Foundations for Decoupled Dynamics in Neural Network Training](https://mlsanity.com/paper/2602.08333/) (2026)
- [When Learning Rates Go Wrong: Early Structural Signals in PPO Actor-Critic](https://mlsanity.com/paper/2603.09950/) (2026)
- [GLAI: GreenLightningAI for Accelerated Training through Knowledge Decoupling](https://mlsanity.com/paper/2510.00883/) (2025)
- [OUIDecay: Adaptive Layer-wise Weight Decay for CNNs Using Online Activation Patterns](https://mlsanity.com/paper/2605.10161/) (2026)
- [OUI as a Structural Observable: Towards an Activation-Centric View of Neural Network Training](https://mlsanity.com/paper/2605.11570/) (2026)
- [FedOUI: OUI-Guided Client Weighting for Federated Aggregation](https://mlsanity.com/paper/2605.11571/) (2026)

## Abstract (arXiv)

We introduce the Overfitting-Underfitting Indicator (OUI), a novel tool for monitoring the training dynamics of Deep Neural Networks (DNNs) and identifying optimal regularization hyperparameters. Specifically, we validate that OUI can effectively guide the selection of the Weight Decay (WD) hyperparameter by indicating whether a model is overfitting or underfitting during training without requiring validation data. Through experiments on DenseNet-BC-100 with CIFAR- 100, EfficientNet-B0 with TinyImageNet and ResNet-34 with ImageNet-1K, we show that maintaining OUI within a prescribed interval correlates strongly with improved generalization and validation scores. Notably, OUI converges significantly faster than traditional metrics such as loss or accuracy, enabling practitioners to identify optimal WD (hyperparameter) values within the early stages of training. By leveraging OUI as a reliable indicator, we can determine early in training whether the chosen WD value leads the model to underfit the training data, overfit, or strike a well-balanced trade-off that maximizes validation scores. This enables more precise WD tuning for optimal performance on the tested datasets and DNNs. All code for reproducing these experiments is available at https://github.com/AlbertoFdezHdez/OUI.