# Data Dropout: Optimizing Training Data for Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1809.00193 · HTML: https://mlsanity.com/paper/1809.00193/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Sep 2018.

## Built on by (most cited first, 23 of 23)

- [Scaling Up Influence Functions](https://mlsanity.com/paper/2112.03052/) (2021)
- [Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs](https://mlsanity.com/paper/2204.12013/) (2022)
- [Exploring Example Influence in Continual Learning](https://mlsanity.com/paper/2209.12241/) (2022)
- [Rethinking Influence Functions of Neural Networks in the Over-parameterized Regime](https://mlsanity.com/paper/2112.08297/) (2021)
- [Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics](https://mlsanity.com/paper/2209.10015/) (2022)
- [Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association](https://mlsanity.com/paper/2103.07293/) (2021)
- [Regularizing Second-Order Influences for Continual Learning](https://mlsanity.com/paper/2304.10177/) (2023)
- [ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation](https://mlsanity.com/paper/2405.14009/) (2024)
- [Data Summarization via Bilevel Optimization](https://mlsanity.com/paper/2109.12534/) (2021)
- [DIWIFT: Discovering Instance-wise Influential Features for Tabular Data](https://mlsanity.com/paper/2207.02773/) (2022)
- [A Survey of Dataset Refinement for Problems in Computer Vision Datasets](https://mlsanity.com/paper/2210.11717/) (2022)
- [You Only Transfer What You Share: Intersection-Induced Graph Transfer Learning for Link Prediction](https://mlsanity.com/paper/2302.14189/) (2023)
- [Progressive Data Dropout: An Embarrassingly Simple Approach to Faster Training](https://mlsanity.com/paper/2505.22342/) (2025)
- [A Simple Remedy for Dataset Bias via Self-Influence: A Mislabeled Sample Perspective](https://mlsanity.com/paper/2411.00360/) (2024)
- [Efficient Neural Architecture Search with Performance Prediction](https://mlsanity.com/paper/2108.01854/) (2021)
- [InfFeed: Influence Functions as a Feedback to Improve the Performance of Subjective Tasks](https://mlsanity.com/paper/2402.14702/) (2024)
- [Adaptive Data Dropout: Towards Self-Regulated Learning in Deep Neural Networks](https://mlsanity.com/paper/2604.12945/) (2026)
- [Interpretable Distance Metric Learning for Handwritten Chinese Character Recognition](https://mlsanity.com/paper/2103.09714/) (2021)
- [Convolutional Neural Network(CNN/ConvNet) in Stock Price Movement Prediction](https://mlsanity.com/paper/2106.01920/) (2021)
- [Unified Regularity Measures for Sample-wise Learning and Generalization](https://mlsanity.com/paper/2108.03913/) (2021)
- [Mistake gating leads to energy and memory efficient continual learning](https://mlsanity.com/paper/2604.14336/) (2026)
- [Using Explainability as a Training-Time Reliability Signal for Efficient ECG Classification](https://mlsanity.com/paper/2606.12252/) (2026)
- [Robust Trajectory Distillation: Hybrid Reweighting Meets Teacher-Inspired Targets](https://mlsanity.com/paper/2606.29837/) (2026)

## Abstract (arXiv)

Deep learning models learn to fit training data while they are highly expected to generalize well to testing data. Most works aim at finding such models by creatively designing architectures and fine-tuning parameters. To adapt to particular tasks, hand-crafted information such as image prior has also been incorporated into end-to-end learning. However, very little progress has been made on investigating how an individual training sample will influence the generalization ability of a model. In other words, to achieve high generalization accuracy, do we really need all the samples in a training dataset? In this paper, we demonstrate that deep learning models such as convolutional neural networks may not favor all training samples, and generalization accuracy can be further improved by dropping those unfavorable samples. Specifically, the influence of removing a training sample is quantifiable, and we propose a Two-Round Training approach, aiming to achieve higher generalization accuracy. We locate unfavorable samples after the first round of training, and then retrain the model from scratch with the reduced training dataset in the second round. Since our approach is essentially different from fine-tuning or further training, the computational cost should not be a concern. Our extensive experimental results indicate that, with identical settings, the proposed approach can boost performance of the well-known networks on both high-level computer vision problems such as image classification, and low-level vision problems such as image denoising.