# Keeping Deep Learning Models in Check: A History-Based Approach to Mitigate Overfitting

arXiv: https://arxiv.org/abs/2401.10359 · HTML: https://mlsanity.com/paper/2401.10359/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 94 references that are in the map.
- First posted on arXiv 18 Jan 2024.

## Builds on (most cited first, 28 of 28)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Overfitting in adversarially robust deep learning](https://mlsanity.com/paper/2002.11569/) (2020)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [A Random Forest Guided Tour](https://mlsanity.com/paper/1511.05741/) (2015)
- [An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation](https://mlsanity.com/paper/1812.08693/) (2018)
- [Regularization for Deep Learning: A Taxonomy](https://mlsanity.com/paper/1710.10686/) (2017)
- [Understanding Catastrophic Overfitting in Single-step Adversarial Training](https://mlsanity.com/paper/2010.01799/) (2020)
- [A Time Series Forest for Classification and Feature Extraction](https://mlsanity.com/paper/1302.2277/) (2013)
- [Multi-task Learning based Pre-trained Language Model for Code Completion](https://mlsanity.com/paper/2012.14631/) (2020)
- [A Survey on Deep Learning for Software Engineering](https://mlsanity.com/paper/2011.14597/) (2020)
- [Retrieve and Refine: Exemplar-based Neural Comment Generation](https://mlsanity.com/paper/1910.10419/) (2019)
- [A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research](https://mlsanity.com/paper/2009.06520/) (2020)
- [Low Curvature Activations Reduce Overfitting in Adversarial Training](https://mlsanity.com/paper/2102.07861/) (2021)
- [Learning Curves for Decision Making in Supervised Machine Learning: A Survey](https://mlsanity.com/paper/2201.12150/) (2022)
- [Small Data, Big Decisions: Model Selection in the Small-Data Regime](https://mlsanity.com/paper/2009.12583/) (2020)
- [Dropout Reduces Underfitting](https://mlsanity.com/paper/2303.01500/) (2023)
- [Easy over Hard: A Case Study on Deep Learning](https://mlsanity.com/paper/1703.00133/) (2017)
- [Random Vector Functional Link Neural Network based Ensemble Deep Learning](https://mlsanity.com/paper/1907.00350/) (2019)
- [Learning Curves for Analysis of Deep Networks](https://mlsanity.com/paper/2010.11029/) (2020)
- [Detecting Overfitting via Adversarial Examples](https://mlsanity.com/paper/1903.02380/) (2019)
- [A deep learning model for estimating story points](https://mlsanity.com/paper/1609.00489/) (2016)
- [NeuRecover: Regression-Controlled Repair of Deep Neural Networks with Training History](https://mlsanity.com/paper/2203.00191/) (2022)
- [Circuit-Based Intrinsic Methods to Detect Overfitting](https://mlsanity.com/paper/1907.01991/) (2019)
- [Early Detection of Promoted Campaigns on Social Media](https://mlsanity.com/paper/1703.07518/) (2017)
- [Model Validation Using Mutated Training Labels: An Exploratory Study](https://mlsanity.com/paper/1905.10201/) (2019)
- [Deep Learning Anti-patterns from Code Metrics History](https://mlsanity.com/paper/1910.07658/) (2019)
- [Cross-Dataset Design Discussion Mining](https://mlsanity.com/paper/2001.01424/) (2020)

## Built on by (most cited first, 6 of 6)

- [Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions](https://mlsanity.com/paper/2503.08979/) (2025)
- [AdaAugment: A Tuning-Free and Adaptive Approach to Enhance Data Augmentation](https://mlsanity.com/paper/2405.11467/) (2024)
- [DG-PPU: Dynamical Graphs based Post-processing of Point Clouds extracted from Knee Ultrasounds](https://mlsanity.com/paper/2411.08926/) (2024)
- [Prevention of Overfitting on Mesh-Structured Data Regressions with a Modified Laplace Operator](https://mlsanity.com/paper/2507.06631/) (2025)
- [Deep learning and abstractive summarisation for radiological reports: an empirical study for adapting the PEGASUS models' family with scarce data](https://mlsanity.com/paper/2509.15419/) (2025)
- [Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning](https://mlsanity.com/paper/2607.24996/) (2026)

## Abstract (arXiv)

In software engineering, deep learning models are increasingly deployed for critical tasks such as bug detection and code review. However, overfitting remains a challenge that affects the quality, reliability, and trustworthiness of software systems that utilize deep learning models. Overfitting can be (1) prevented (e.g., using dropout or early stopping) or (2) detected in a trained model (e.g., using correlation-based approaches). Both overfitting detection and prevention approaches that are currently used have constraints (e.g., requiring modification of the model structure, and high computing resources). In this paper, we propose a simple, yet powerful approach that can both detect and prevent overfitting based on the training history (i.e., validation losses). Our approach first trains a time series classifier on training histories of overfit models. This classifier is then used to detect if a trained model is overfit. In addition, our trained classifier can be used to prevent overfitting by identifying the optimal point to stop a model's training. We evaluate our approach on its ability to identify and prevent overfitting in real-world samples. We compare our approach against correlation-based detection approaches and the most commonly used prevention approach (i.e., early stopping). Our approach achieves an F1 score of 0.91 which is at least 5% higher than the current best-performing non-intrusive overfitting detection approach. Furthermore, our approach can stop training to avoid overfitting at least 32% of the times earlier than early stopping and has the same or a better rate of returning the best model.