# Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting

arXiv: https://arxiv.org/abs/2011.06220 · HTML: https://mlsanity.com/paper/2011.06220/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2020.

## Built on by (most cited first, 28 of 28)

- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations](https://mlsanity.com/paper/2110.12088/) (2021)
- [On-the-fly Modulation for Balanced Multimodal Learning](https://mlsanity.com/paper/2410.11582/) (2024)
- [On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications](https://mlsanity.com/paper/2110.03128/) (2021)
- [Active Neural Mapping](https://mlsanity.com/paper/2308.16246/) (2023)
- [Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization](https://mlsanity.com/paper/2103.17182/) (2021)
- [Sparse Double Descent: Where Network Pruning Aggravates Overfitting](https://mlsanity.com/paper/2206.08684/) (2022)
- [S3IM: Stochastic Structural SIMilarity and Its Unreasonable Effectiveness for Neural Fields](https://mlsanity.com/paper/2308.07032/) (2023)
- [On the Overlooked Structure of Stochastic Gradients](https://mlsanity.com/paper/2212.02083/) (2022)
- [Analyzing Lottery Ticket Hypothesis from PAC-Bayesian Theory Perspective](https://mlsanity.com/paper/2205.07320/) (2022)
- [Lifelong Ensemble Learning based on Multiple Representations for Few-Shot Object Recognition](https://mlsanity.com/paper/2205.01982/) (2022)
- [Towards Practical Tool Usage for Continually Learning LLMs](https://mlsanity.com/paper/2404.09339/) (2024)
- [TAG: Task-based Accumulated Gradients for Lifelong learning](https://mlsanity.com/paper/2105.05155/) (2021)
- [SplitNet: Learnable Clean-Noisy Label Splitting for Learning with Noisy Labels](https://mlsanity.com/paper/2211.11753/) (2022)
- [Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models](https://mlsanity.com/paper/2412.03587/) (2024)
- [On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvature](https://mlsanity.com/paper/2602.05600/) (2026)
- [Deep learning, stochastic gradient descent and diffusion maps](https://mlsanity.com/paper/2204.01365/) (2022)
- [Goldilocks-curriculum Domain Randomization and Fractal Perlin Noise with Application to Sim2Real Pneumonia Lesion Detection](https://mlsanity.com/paper/2204.13849/) (2022)
- [Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Better](https://mlsanity.com/paper/2602.05393/) (2026)
- [Intrinsic-Noise Consolidation: A Doob-Barrier-Conditioned Diffusion Turns Analog Device Noise into a Continual-Learning Resource](https://mlsanity.com/paper/2607.06924/) (2026)
- [The Role of Bio-Inspired Modularity in General Learning](https://mlsanity.com/paper/2109.15097/) (2021)
- [Review of Kernel Learning for Intra-Hour Solar Forecasting with Infrared Sky Images and Cloud Dynamic Feature Extraction](https://mlsanity.com/paper/2110.05622/) (2021)
- [Research Trends and Applications of Data Augmentation Algorithms](https://mlsanity.com/paper/2207.08817/) (2022)
- [Feature Noise Boosts DNN Generalization under Label Noise](https://mlsanity.com/paper/2308.01609/) (2023)
- [Neural Field Classifiers via Target Encoding and Classification Loss](https://mlsanity.com/paper/2403.01058/) (2024)
- [Provably Robust Pre-Trained Ensembles for Biomarker-Based Cancer Classification](https://mlsanity.com/paper/2406.10087/) (2024)
- [Learning from Ambiguous Data with Hard Labels](https://mlsanity.com/paper/2501.01844/) (2025)
- [On the Effectiveness of Membership Inference in Targeted Data Extraction from Large Language Models](https://mlsanity.com/paper/2512.13352/) (2025)

## Abstract (arXiv)

Deep learning is often criticized by two serious issues which rarely exist in natural nervous systems: overfitting and catastrophic forgetting. It can even memorize randomly labelled data, which has little knowledge behind the instance-label pairs. When a deep network continually learns over time by accommodating new tasks, it usually quickly overwrites the knowledge learned from previous tasks. Referred to as the it neural variability, it is well-known in neuroscience that human brain reactions exhibit substantial variability even in response to the same stimulus. This mechanism balances accuracy and plasticity/flexibility in the motor learning of natural nervous systems. Thus it motivates us to design a similar mechanism named it artificial neural variability (ANV), which helps artificial neural networks learn some advantages from ``natural'' neural networks. We rigorously prove that ANV plays as an implicit regularizer of the mutual information between the training data and the learned model. This result theoretically guarantees ANV a strictly improved generalizability, robustness to label noise, and robustness to catastrophic forgetting. We then devise a it neural variable risk minimization (NVRM) framework and it neural variable optimizers to achieve ANV for conventional network architectures in practice. The empirical studies demonstrate that NVRM can effectively relieve overfitting, label noise memorization, and catastrophic forgetting at negligible costs. footnoteCode: urlhttps://github.com/zeke-xie/artificial-neural-variability-for-deep-learning.