# DisturbLabel: Regularizing CNN on the Loss Layer

arXiv: https://arxiv.org/abs/1605.00055 · HTML: https://mlsanity.com/paper/1605.00055/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2016.

## Built on by (most cited first, 30 of 41)

- [The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration](https://mlsanity.com/paper/2111.15430/) (2021)
- [Open-set Label Noise Can Improve Robustness Against Inherent Label Noise](https://mlsanity.com/paper/2106.10891/) (2021)
- [Salient Objects in Clutter](https://mlsanity.com/paper/2105.03053/) (2021)
- [Information-Theoretic Bias Reduction via Causal View of Spurious Correlation](https://mlsanity.com/paper/2201.03121/) (2022)
- [On the Privacy Effect of Data Enhancement via the Lens of Memorization](https://mlsanity.com/paper/2208.08270/) (2022)
- [Adaptive Label Smoothing To Regularize Large-Scale Graph Training](https://mlsanity.com/paper/2108.13555/) (2021)
- [Iterative Teaching by Label Synthesis](https://mlsanity.com/paper/2110.14432/) (2021)
- [Dense Uncertainty Estimation](https://mlsanity.com/paper/2110.06427/) (2021)
- [GeoPointGAN: Synthetic Spatial Data with Local Label Differential Privacy](https://mlsanity.com/paper/2205.08886/) (2022)
- [NoiLIn: Improving Adversarial Training and Correcting Stereotype of Noisy Labels](https://mlsanity.com/paper/2105.14676/) (2021)
- [Relaxed Quantile Regression: Prediction Intervals for Asymmetric Noise](https://mlsanity.com/paper/2406.03258/) (2024)
- [DynamicKD: An Effective Knowledge Distillation via Dynamic Entropy Correction-Based Distillation for Gap Optimizing](https://mlsanity.com/paper/2305.05233/) (2023)
- [ProSelfLC: Progressive Self Label Correction Towards A Low-Temperature Entropy State](https://mlsanity.com/paper/2207.00118/) (2022)
- [Enhancing Sample Utilization through Sample Adaptive Augmentation in Semi-Supervised Learning](https://mlsanity.com/paper/2309.03598/) (2023)
- [Predicting Out-of-Domain Generalization with Neighborhood Invariance](https://mlsanity.com/paper/2207.02093/) (2022)
- [Class-Level Logit Perturbation](https://mlsanity.com/paper/2209.05668/) (2022)
- [Rethinking Class Imbalance in Machine Learning](https://mlsanity.com/paper/2305.03900/) (2023)
- [Midpoint Regularization: from High Uncertainty Training to Conservative Classification](https://mlsanity.com/paper/2106.13913/) (2021)
- [Model Calibration in Dense Classification with Adaptive Label Perturbation](https://mlsanity.com/paper/2307.13539/) (2023)
- [DEAL-300K: Diffusion-based Editing Area Localization with a 300K-Scale Dataset and Frequency-Prompted Baseline](https://mlsanity.com/paper/2511.23377/) (2025)
- [Alleviating Noisy-label Effects in Image Classification via Probability Transition Matrix](https://mlsanity.com/paper/2110.08866/) (2021)
- [Generalised Co-Salient Object Detection](https://mlsanity.com/paper/2208.09668/) (2022)
- [Implicit Counterfactual Data Augmentation for Robust Learning](https://mlsanity.com/paper/2304.13431/) (2023)
- [Mitigating Shortcuts in Language Models with Soft Label Encoding](https://mlsanity.com/paper/2309.09380/) (2023)
- [Temporal Difference Learning with Constrained Initial Representations](https://mlsanity.com/paper/2602.11800/) (2026)
- [Label Smoothed Embedding Hypothesis for Out-of-Distribution Detection](https://mlsanity.com/paper/2102.05131/) (2021)
- [Bias Loss for Mobile Neural Networks](https://mlsanity.com/paper/2107.11170/) (2021)
- [Context-based Virtual Adversarial Training for Text Classification with Noisy Labels](https://mlsanity.com/paper/2206.11851/) (2022)
- [Confidence-Aware Paced-Curriculum Learning by Label Smoothing for Surgical Scene Understanding](https://mlsanity.com/paper/2212.11511/) (2022)
- [Keep It Simple: CNN Model Complexity Studies for Interference Classification Tasks](https://mlsanity.com/paper/2303.03326/) (2023)

## Abstract (arXiv)

During a long period of time we are combating over-fitting in the CNN training process with model regularization, including weight decay, model averaging, data augmentation, etc. In this paper, we present DisturbLabel, an extremely simple algorithm which randomly replaces a part of labels as incorrect values in each iteration. Although it seems weird to intentionally generate incorrect training labels, we show that DisturbLabel prevents the network training from over-fitting by implicitly averaging over exponentially many networks which are trained with different label sets. To the best of our knowledge, DisturbLabel serves as the first work which adds noises on the loss layer. Meanwhile, DisturbLabel cooperates well with Dropout to provide complementary regularization functions. Experiments demonstrate competitive recognition results on several popular image recognition datasets.