# Overfitting Mechanism and Avoidance in Deep Neural Networks

arXiv: https://arxiv.org/abs/1901.06566 · HTML: https://mlsanity.com/paper/1901.06566/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jan 2019.

## Built on by (most cited first, 30 of 30)

- [Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Prediction Models](https://mlsanity.com/paper/2209.06053/) (2022)
- [Interpreting the Curse of Dimensionality from Distance Concentration and Manifold Effect](https://mlsanity.com/paper/2401.00422/) (2023)
- [Unveiling Over-Memorization in Finetuning LLMs for Reasoning Tasks](https://mlsanity.com/paper/2508.04117/) (2025)
- [Hybrid Deep Learning Model using SPCAGAN Augmentation for Insider Threat Analysis](https://mlsanity.com/paper/2203.02855/) (2022)
- [GenEFT: Understanding Statics and Dynamics of Model Generalization via Effective Theory](https://mlsanity.com/paper/2402.05916/) (2024)
- [Semi-Supervised Object Detection: A Survey on Progress from CNN to Transformer](https://mlsanity.com/paper/2407.08460/) (2024)
- [DCNet: A Data-Driven Framework for DVL Calibration](https://mlsanity.com/paper/2410.08809/) (2024)
- [Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations](https://mlsanity.com/paper/2211.08794/) (2022)
- [Multi-Epoch Learning for Deep Click-Through Rate Prediction Models](https://mlsanity.com/paper/2305.19531/) (2023)
- [What can we learn from misclassified ImageNet images?](https://mlsanity.com/paper/2201.08098/) (2022)
- [TreeSketchNet: From Sketch To 3D Tree Parameters Generation](https://mlsanity.com/paper/2207.12297/) (2022)
- [United We Stand: Using Epoch-wise Agreement of Ensembles to Combat Overfit](https://mlsanity.com/paper/2310.11077/) (2023)
- [PromptFix: Few-shot Backdoor Removal via Adversarial Prompt Tuning](https://mlsanity.com/paper/2406.04478/) (2024)
- [Multi-Epoch learning with Data Augmentation for Deep Click-Through Rate Prediction](https://mlsanity.com/paper/2407.01607/) (2024)
- [Constrained Sliced Wasserstein Embedding](https://mlsanity.com/paper/2506.02203/) (2025)
- [XPASC: Measuring Generalization in Weak Supervision by Explainability and Association](https://mlsanity.com/paper/2206.01444/) (2022)
- [Embarrassingly Simple MixUp for Time-series](https://mlsanity.com/paper/2304.04271/) (2023)
- [Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions in LSTM Networks](https://mlsanity.com/paper/2505.20030/) (2025)
- [Classification of multivariate weakly-labelled time-series with attention](https://mlsanity.com/paper/2102.08245/) (2021)
- [Research Trends and Applications of Data Augmentation Algorithms](https://mlsanity.com/paper/2207.08817/) (2022)
- [Realization of Causal Representation Learning to Adjust Confounding Bias in Latent Space](https://mlsanity.com/paper/2211.08573/) (2022)
- [Geometric Perception based Efficient Text Recognition](https://mlsanity.com/paper/2302.03873/) (2023)
- [Intelligent Chemical Purification Technique Based on Machine Learning](https://mlsanity.com/paper/2404.09114/) (2024)
- [Sharpness-Aware Minimization with Z-Score Gradient Filtering](https://mlsanity.com/paper/2505.02369/) (2025)
- [Experimental Study on Automatically Assembling Custom Catering Packages With a 3-DOF Delta Robot Using Deep Learning Methods](https://mlsanity.com/paper/2505.11879/) (2025)
- [Exploring Graph Representations of Logical Forms for Language Modeling](https://mlsanity.com/paper/2505.14523/) (2025)
- [Backbone Augmented Training for Adaptations](https://mlsanity.com/paper/2506.04288/) (2025)
- [Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models](https://mlsanity.com/paper/2510.00563/) (2025)
- [Neural-Assisted in-Motion Self-Heading Alignment](https://mlsanity.com/paper/2604.00168/) (2026)
- [Misalignment of Low-Loss Regions Causes Grokking](https://mlsanity.com/paper/2610.00620/) (2026)

## Abstract (arXiv)

Assisted by the availability of data and high performance computing, deep learning techniques have achieved breakthroughs and surpassed human performance empirically in difficult tasks, including object recognition, speech recognition, and natural language processing. As they are being used in critical applications, understanding underlying mechanisms for their successes and limitations is imperative. In this paper, we show that overfitting, one of the fundamental issues in deep neural networks, is due to continuous gradient updating and scale sensitiveness of cross entropy loss. By separating samples into correctly and incorrectly classified ones, we show that they behave very differently, where the loss decreases in the correct ones and increases in the incorrect ones. Furthermore, by analyzing dynamics during training, we propose a consensus-based classification algorithm that enables us to avoid overfitting and significantly improve the classification accuracy especially when the number of training samples is limited. As each trained neural network depends on extrinsic factors such as initial values as well as training data, requiring consensus among multiple models reduces extrinsic factors substantially; for statistically independent models, the reduction is exponential. Compared to ensemble algorithms, the proposed algorithm avoids overgeneralization by not classifying ambiguous inputs. Systematic experimental results demonstrate the effectiveness of the proposed algorithm. For example, using only 1000 training samples from MNIST dataset, the proposed algorithm achieves 95% accuracy, significantly higher than any of the individual models, with 90% of the test samples classified.