# Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence

arXiv: https://arxiv.org/abs/1905.13277 · HTML: https://mlsanity.com/paper/1905.13277/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2019.

## Built on by (most cited first, 30 of 54)

- [High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation](https://mlsanity.com/paper/2205.01445/) (2022)
- [MobileOne: An Improved One millisecond Mobile Backbone](https://mlsanity.com/paper/2206.04040/) (2022)
- [Consensus Control for Decentralized Deep Learning](https://mlsanity.com/paper/2102.04828/) (2021)
- [A study on the plasticity of neural networks](https://mlsanity.com/paper/2106.00042/) (2021)
- [Properties of the After Kernel](https://mlsanity.com/paper/2105.10585/) (2021)
- [A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learning](https://mlsanity.com/paper/2210.04561/) (2022)
- [SoMA: Singular Value Decomposed Minor Components Adaptation for Domain Generalizable Representation Learning](https://mlsanity.com/paper/2412.04077/) (2024)
- [Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition](https://mlsanity.com/paper/2503.18595/) (2025)
- [DivAug: Plug-in Automated Data Augmentation with Explicit Diversity Maximization](https://mlsanity.com/paper/2103.14545/) (2021)
- [Critical Learning Periods for Multisensory Integration in Deep Networks](https://mlsanity.com/paper/2210.04643/) (2022)
- [Over-training with Mixup May Hurt Generalization](https://mlsanity.com/paper/2303.01475/) (2023)
- [Tangent Transformers for Composition, Privacy and Removal](https://mlsanity.com/paper/2307.08122/) (2023)
- [Critical Learning Periods Emerge Even in Deep Linear Networks](https://mlsanity.com/paper/2308.12221/) (2023)
- [The activity-weight duality in feed forward neural networks: The geometric determinants of generalization](https://mlsanity.com/paper/2203.10736/) (2022)
- [Edge of chaos as a guiding principle for modern neural network training](https://mlsanity.com/paper/2107.09437/) (2021)
- [An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descent](https://mlsanity.com/paper/2310.11291/) (2023)
- [Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review](https://mlsanity.com/paper/2311.11883/) (2023)
- [Neural Architecture Search as Program Transformation Exploration](https://mlsanity.com/paper/2102.06599/) (2021)
- [Critical Learning Periods in Federated Learning](https://mlsanity.com/paper/2109.05613/) (2021)
- [Cyclical Focal Loss](https://mlsanity.com/paper/2202.08978/) (2022)
- [FUN with Fisher: Improving Generalization of Adapter-Based Cross-lingual Transfer with Scheduled Unfreezing](https://mlsanity.com/paper/2301.05487/) (2023)
- [Exploring Weight Balancing on Long-Tailed Recognition Problem](https://mlsanity.com/paper/2305.16573/) (2023)
- [Intraclass clustering: an implicit learning ability that regularizes DNNs](https://mlsanity.com/paper/2103.06733/) (2021)
- [One-Cycle Pruning: Pruning ConvNets Under a Tight Training Budget](https://mlsanity.com/paper/2107.02086/) (2021)
- [General Cyclical Training of Neural Networks](https://mlsanity.com/paper/2202.08835/) (2022)
- [Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Rates](https://mlsanity.com/paper/2206.00832/) (2022)
- [On the special role of class-selective neurons in early training](https://mlsanity.com/paper/2305.17409/) (2023)
- [Data-Efficient Augmentation for Training Neural Networks](https://mlsanity.com/paper/2210.08363/) (2022)
- [Efficient Scheduling of Data Augmentation for Deep Reinforcement Learning](https://mlsanity.com/paper/2102.08581/) (2021)
- [Reduce, Reuse, Recycle: Improving Training Efficiency with Distillation](https://mlsanity.com/paper/2211.00683/) (2022)

## Abstract (arXiv)

Regularization is typically understood as improving generalization by altering the landscape of local extrema to which the model eventually converges. Deep neural networks (DNNs), however, challenge this view: We show that removing regularization after an initial transient period has little effect on generalization, even if the final loss landscape is the same as if there had been no regularization. In some cases, generalization even improves after interrupting regularization. Conversely, if regularization is applied only after the initial transient, it has no effect on the final solution, whose generalization gap is as bad as if regularization never happened. This suggests that what matters for training deep networks is not just whether or how, but when to regularize. The phenomena we observe are manifest in different datasets (CIFAR-10, CIFAR-100), different architectures (ResNet-18, All-CNN), different regularization methods (weight decay, data augmentation), different learning rate schedules (exponential, piece-wise constant). They collectively suggest that there is a ``critical period'' for regularizing deep networks that is decisive of the final performance. More analysis should, therefore, focus on the transient rather than asymptotic behavior of learning.