# ShakeDrop Regularization for Deep Residual Learning

arXiv: https://arxiv.org/abs/1802.02375 · HTML: https://mlsanity.com/paper/1802.02375/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2018.

## Built on by (most cited first, 30 of 40)

- [ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks](https://mlsanity.com/paper/2102.11600/) (2021)
- [Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learning](https://mlsanity.com/paper/2202.03599/) (2022)
- [Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks](https://mlsanity.com/paper/2201.03299/) (2022)
- [Spectral Bias in Practice: The Role of Function Frequency in Generalization](https://mlsanity.com/paper/2110.02424/) (2021)
- [TeachAugment: Data Augmentation Optimization Using Teacher Knowledge](https://mlsanity.com/paper/2202.12513/) (2022)
- [Bootstrap Generalization Ability from Loss Landscape Perspective](https://mlsanity.com/paper/2209.08473/) (2022)
- [DivAug: Plug-in Automated Data Augmentation with Explicit Diversity Maximization](https://mlsanity.com/paper/2103.14545/) (2021)
- [Saliency Grafting: Innocuous Attribution-Guided Mixup with Calibrated Label Mixing](https://mlsanity.com/paper/2112.08796/) (2021)
- [MutualNet: Adaptive ConvNet via Mutual Learning from Different Model Configurations](https://mlsanity.com/paper/2105.07085/) (2021)
- [Omnizart: A General Toolbox for Automatic Music Transcription](https://mlsanity.com/paper/2106.00497/) (2021)
- [CAM-loss: Towards Learning Spatially Discriminative Feature Representations](https://mlsanity.com/paper/2109.01359/) (2021)
- [Improving Robustness with Adaptive Weight Decay](https://mlsanity.com/paper/2210.00094/) (2022)
- [When to Learn What: Model-Adaptive Data Augmentation Curriculum](https://mlsanity.com/paper/2309.04747/) (2023)
- [A Survey of Automated Data Augmentation Algorithms for Deep Learning-based Image Classification Tasks](https://mlsanity.com/paper/2206.06544/) (2022)
- [Finding Optimal Tangent Points for Reducing Distortions of Hard-label Attacks](https://mlsanity.com/paper/2111.07492/) (2021)
- [Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey](https://mlsanity.com/paper/2405.01725/) (2024)
- [Privacy-Preserving Image Classification Using Vision Transformer](https://mlsanity.com/paper/2205.12041/) (2022)
- [Exploring Temporally Dynamic Data Augmentation for Video Recognition](https://mlsanity.com/paper/2206.15015/) (2022)
- [Improving Model Generalization by Agreement of Learned Representations from Data Augmentation](https://mlsanity.com/paper/2110.10536/) (2021)
- [AdvLogo: Adversarial Patch Attack against Object Detectors based on Diffusion Models](https://mlsanity.com/paper/2409.07002/) (2024)
- [DAAS: Differentiable Architecture and Augmentation Policy Search](https://mlsanity.com/paper/2109.15273/) (2021)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [FocusedDropout for Convolutional Neural Network](https://mlsanity.com/paper/2103.15425/) (2021)
- [Color-NeuraCrypt: Privacy-Preserving Color-Image Classification Using Extended Random Neural Networks](https://mlsanity.com/paper/2301.04875/) (2023)
- [Deep Neural Networks Inspired by Differential Equations](https://mlsanity.com/paper/2510.09685/) (2025)
- [ContextMix: A context-aware data augmentation method for industrial visual inspection systems](https://mlsanity.com/paper/2401.10050/) (2024)
- [MaskBlur: Spatial and Angular Data Augmentation for Light Field Image Super-Resolution](https://mlsanity.com/paper/2410.06478/) (2024)
- [Boosting Ray Search Procedure of Hard-label Attacks with Transfer-based Priors](https://mlsanity.com/paper/2507.17577/) (2025)
- [Deep Neural Network Calibration by Reducing Classifier Shift with Stochastic Masking](https://mlsanity.com/paper/2508.09116/) (2025)
- [IPF-RDA: An Information-Preserving Framework for Robust Data Augmentation](https://mlsanity.com/paper/2509.16678/) (2025)

## Abstract (arXiv)

Overfitting is a crucial problem in deep neural networks, even in the latest network architectures. In this paper, to relieve the overfitting effect of ResNet and its improvements (i.e., Wide ResNet, PyramidNet, and ResNeXt), we propose a new regularization method called ShakeDrop regularization. ShakeDrop is inspired by Shake-Shake, which is an effective regularization method, but can be applied to ResNeXt only. ShakeDrop is more effective than Shake-Shake and can be applied not only to ResNeXt but also ResNet, Wide ResNet, and PyramidNet. An important key is to achieve stability of training. Because effective regularization often causes unstable training, we introduce a training stabilizer, which is an unusual use of an existing regularizer. Through experiments under various conditions, we demonstrate the conditions under which ShakeDrop works well.