No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasets
Not on the map. Why?
Abstract (arXiv)
Solving image classification tasks given small training datasets remains an open challenge for modern computer vision. Aggressive data augmentation and generative models are among the most straightforward approaches to overcoming the lack of data. However, the first fails to be agnostic to varying image domains, while the latter requires additional compute and careful design. In this work, we study alternative regularization strategies to push the limits of supervised learning on small image classification datasets. In particular, along with the model size and training schedule scaling, we employ a heuristic to select (semi) optimal learning rate and weight decay couples via the norm of model parameters. By training on only 1% of the original CIFAR-10 training set (i.e., 50 images per class) and testing on ciFAIR-10, a variant of the original CIFAR without duplicated images, we reach a test accuracy of 66.5%, on par with the best state-of-the-art methods.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 60 of its 82 references that are in the map. First posted on arXiv 4 Sep 2023.
Builds on
60 of 82 references.
References · 60
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
- 2016Wide Residual Networkscited by 3,057
- 2017A Closer Look at Memorization in Deep Networkscited by 936
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
- 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
38 more
- 2018Deep Visual Domain Adaptation: A Surveycited by 634
- 2021ImageNet-21K Pretraining for the Massescited by 607
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
- 2018Do Better ImageNet Models Transfer Better?cited by 560
- 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
- 2019Generalizing from a Few Examples: A Survey on Few-Shot Learningcited by 462
- 2018Rethinking ImageNet Pre-trainingcited by 376
- 2016Harmonic Networks: Deep Translation and Rotation Equivariancecited by 293
- 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
- 2017High-dimensional dynamics of generalization error in neural networkscited by 238
- 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
- 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
- 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
- 2020On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Locationcited by 141
- 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
- 2018Three Mechanisms of Weight Decay Regularizationcited by 112
- 2022The Effects of Regularization and Data Augmentation are Class Dependentcited by 67
- 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
- 2019Do We Train on Test Data? Purging CIFAR of Near-Duplicatescited by 61
- 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
- 2019Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergencecited by 54
- 2020A Close Look at Deep Learning with Small Datacited by 54
- 2020Auxiliary Learning by Implicit Differentiationcited by 45
- 2020A Brief Prehistory of Double Descentcited by 44
- 2019Deep Learning on Small Datasets without Pre-Training using Cosine Losscited by 41
- 2017OL'E: Orthogonal Low-rank Embedding, A Plug and Play Geometric Loss for Deep Learningcited by 37
- 2017Scaling the Scattering Transform: Deep Hybrid Networkscited by 35
- 2018Scattering Networks for Hybrid Representation Learningcited by 33
- 2020Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedulecited by 25
- 2021Multi-scale Feature Learning Dynamics: Insights for Double Descentcited by 24
- 2020Small Data, Big Decisions: Model Selection in the Small-Data Regimecited by 23
- 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
- 2018DADA: Deep Adversarial Data Augmentation for Extremely Low Data Regime Classificationcited by 18
- 2021Parametric Scattering Networkscited by 15
- 2018Guided Dropoutcited by 14
- 2022Equivariance and Invariance Inductive Bias for Learning from Insufficient Datacited by 14
- 2020Distilling Visual Priors from Self-Supervised Learningcited by 13
- 2021VIPriors 1: Visual Inductive Priors for Data-Efficient Deep Learning Challengescited by 11
50 most cited of 60 All 60 (JSON)
Built on by
Citing papers per year: 2024: 1.
1 paper.