# Improving Resnet-9 Generalization Trained on Small Datasets

arXiv: https://arxiv.org/abs/2309.03965 · HTML: https://mlsanity.com/paper/2309.03965/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 18 references that are in the map.
- First posted on arXiv 7 Sep 2023.

## Builds on (most cited first, 7 of 7)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Gradient Centralization: A New Optimization Technique for Deep Neural Networks](https://mlsanity.com/paper/2004.01461/) (2020)
- [Continuously Differentiable Exponential Linear Units](https://mlsanity.com/paper/1704.07483/) (2017)
- [A Short Study on Compressing Decoder-Based Language Models](https://mlsanity.com/paper/2110.08460/) (2021)

## Built on by (most cited first, 2 of 2)

- [Speeding up Resnet Architecture with Layers Targeted Low Rank Decomposition](https://mlsanity.com/paper/2309.12412/) (2023)
- [SwiftLearn: A Data-Efficient Training Method of Deep Learning Models using Importance Sampling](https://mlsanity.com/paper/2311.15134/) (2023)

## Abstract (arXiv)

This paper presents our proposed approach that won the first prize at the ICLR competition on Hardware Aware Efficient Training. The challenge is to achieve the highest possible accuracy in an image classification task in less than 10 minutes. The training is done on a small dataset of 5000 images picked randomly from CIFAR-10 dataset. The evaluation is performed by the competition organizers on a secret dataset with 1000 images of the same size. Our approach includes applying a series of technique for improving the generalization of ResNet-9 including: sharpness aware optimization, label smoothing, gradient centralization, input patch whitening as well as metalearning based training. Our experiments show that the ResNet-9 can achieve the accuracy of 88% while trained only on a 10% subset of CIFAR-10 dataset in less than 10 minuets