# No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasets

arXiv: https://arxiv.org/abs/2309.01694 · HTML: https://mlsanity.com/paper/2309.01694/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 82 references that are in the map.
- First posted on arXiv 4 Sep 2023.

## Builds on (most cited first, 30 of 60)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [Harmonic Networks: Deep Translation and Rotation Equivariance](https://mlsanity.com/paper/1612.04642/) (2016)
- [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [A Constructive Prediction of the Generalization Error Across Scales](https://mlsanity.com/paper/1909.12673/) (2019)
- [Sensitivity and Generalization in Neural Networks: an Empirical Study](https://mlsanity.com/paper/1802.08760/) (2018)
- [Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks](https://mlsanity.com/paper/1907.04595/) (2019)
- [On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Location](https://mlsanity.com/paper/2003.07064/) (2020)
- [L2 Regularization versus Batch and Weight Normalization](https://mlsanity.com/paper/1706.05350/) (2017)
- [Three Mechanisms of Weight Decay Regularization](https://mlsanity.com/paper/1810.12281/) (2018)
- [The Effects of Regularization and Data Augmentation are Class Dependent](https://mlsanity.com/paper/2204.03632/) (2022)
- [Norm matters: efficient and accurate normalization schemes in deep networks](https://mlsanity.com/paper/1803.01814/) (2018)

## Built on by (most cited first, 1 of 1)

- [Twin: Tuning Learning Rate and Weight Decay of Deep Homogeneous Classifiers without Validation](https://mlsanity.com/paper/2403.05532/) (2024)

## Abstract (arXiv)

Solving image classification tasks given small training datasets remains an open challenge for modern computer vision. Aggressive data augmentation and generative models are among the most straightforward approaches to overcoming the lack of data. However, the first fails to be agnostic to varying image domains, while the latter requires additional compute and careful design. In this work, we study alternative regularization strategies to push the limits of supervised learning on small image classification datasets. In particular, along with the model size and training schedule scaling, we employ a heuristic to select (semi) optimal learning rate and weight decay couples via the norm of model parameters. By training on only 1% of the original CIFAR-10 training set (i.e., 50 images per class) and testing on ciFAIR-10, a variant of the original CIFAR without duplicated images, we reach a test accuracy of 66.5%, on par with the best state-of-the-art methods.