# The Theory Behind Overfitting, Cross Validation, Regularization, Bagging, and Boosting: Tutorial

arXiv: https://arxiv.org/abs/1905.12787 · HTML: https://mlsanity.com/paper/1905.12787/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2019.

## Built on by (most cited first, 22 of 22)

- [MTH-IDS: A Multi-Tiered Hybrid Intrusion Detection System for Internet of Vehicles](https://mlsanity.com/paper/2105.13289/) (2021)
- [A review on data-driven constitutive laws for solids](https://mlsanity.com/paper/2405.03658/) (2024)
- [KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Survey](https://mlsanity.com/paper/2110.01858/) (2021)
- [Johnson-Lindenstrauss Lemma, Linear and Nonlinear Random Projections, Random Fourier Features, and Random Kitchen Sinks: Tutorial and Survey](https://mlsanity.com/paper/2108.04172/) (2021)
- [An Analysis on Ensemble Learning optimized Medical Image Classification with Deep Convolutional Neural Networks](https://mlsanity.com/paper/2201.11440/) (2022)
- [Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey](https://mlsanity.com/paper/2107.12521/) (2021)
- [Generative Adversarial Networks and Adversarial Autoencoders: Tutorial and Survey](https://mlsanity.com/paper/2111.13282/) (2021)
- [Performance and Interpretability Comparisons of Supervised Machine Learning Algorithms: An Empirical Study](https://mlsanity.com/paper/2204.12868/) (2022)
- [NetTiSA: Extended IP Flow with Time-series Features for Universal Bandwidth-constrained High-speed Network Traffic Classification](https://mlsanity.com/paper/2310.05530/) (2023)
- [Malware Detection at the Edge with Lightweight LLMs: A Performance Evaluation](https://mlsanity.com/paper/2503.04302/) (2025)
- [Who could be behind QAnon? Authorship attribution with supervised machine-learning](https://mlsanity.com/paper/2303.02078/) (2023)
- [Simmering: Sufficient is better than optimal for training neural networks](https://mlsanity.com/paper/2410.19912/) (2024)
- [Cascade Bagging for Accuracy Prediction with Few Training Samples](https://mlsanity.com/paper/2108.05613/) (2021)
- [Testing for Overfitting](https://mlsanity.com/paper/2305.05792/) (2023)
- [Registration of algebraic varieties using Riemannian optimization](https://mlsanity.com/paper/2401.08562/) (2024)
- [Predict NAS Multi-Task by Stacking Ensemble Models using GP-NAS](https://mlsanity.com/paper/2305.01667/) (2023)
- [Analyses and Concerns in Precision Medicine: A Statistical Perspective](https://mlsanity.com/paper/2401.06899/) (2024)
- [A Computer Vision-Based Quality Assessment Technique for the automatic control of consumables for analytical laboratories](https://mlsanity.com/paper/2404.10454/) (2024)
- [DUA-D2C: Dynamic Uncertainty Aware Method for Overfitting Remediation in Deep Learning](https://mlsanity.com/paper/2411.15876/) (2024)
- [MAP4TS: A Multi-Aspect Prompting Framework for Time-Series Forecasting with Large Language Models](https://mlsanity.com/paper/2510.23090/) (2025)
- [Estimation of aboveground biomass in a tropical dry forest: An intercomparison of airborne, unmanned, and space laser scanning](https://mlsanity.com/paper/2510.27408/) (2025)
- [SX-GeoTree: Self-eXplaining Geospatial Regression Tree Incorporating the Spatial Similarity of Feature Attributions](https://mlsanity.com/paper/2511.19845/) (2025)

## Abstract (arXiv)

In this tutorial paper, we first define mean squared error, variance, covariance, and bias of both random variables and classification/predictor models. Then, we formulate the true and generalization errors of the model for both training and validation/test instances where we make use of the Stein's Unbiased Risk Estimator (SURE). We define overfitting, underfitting, and generalization using the obtained true and generalization errors. We introduce cross validation and two well-known examples which are K-fold and leave-one-out cross validations. We briefly introduce generalized cross validation and then move on to regularization where we use the SURE again. We work on both ℓ2 and ℓ1 norm regularizations. Then, we show that bootstrap aggregating (bagging) reduces the variance of estimation. Boosting, specifically AdaBoost, is introduced and it is explained as both an additive model and a maximum margin model, i.e., Support Vector Machine (SVM). The upper bound on the generalization error of boosting is also provided to show why boosting prevents from overfitting. As examples of regularization, the theory of ridge and lasso regressions, weight decay, noise injection to input/weights, and early stopping are explained. Random forest, dropout, histogram of oriented gradients, and single shot multi-box detector are explained as examples of bagging in machine learning and computer vision. Finally, boosting tree and SVM models are mentioned as examples of boosting.