# Tune It or Don't Use It: Benchmarking Data-Efficient Image Classification

arXiv: https://arxiv.org/abs/2108.13122 · HTML: https://mlsanity.com/paper/2108.13122/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 30 Aug 2021.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning](https://mlsanity.com/paper/1806.06193/) (2018)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Location](https://mlsanity.com/paper/2003.07064/) (2020)
- [Do We Train on Test Data? Purging CIFAR of Near-Duplicates](https://mlsanity.com/paper/1902.00423/) (2019)
- [A Close Look at Deep Learning with Small Data](https://mlsanity.com/paper/2003.12843/) (2020)
- [Deep Learning on Small Datasets without Pre-Training using Cosine Loss](https://mlsanity.com/paper/1901.09054/) (2019)
- [OL'E: Orthogonal Low-rank Embedding, A Plug and Play Geometric Loss for Deep Learning](https://mlsanity.com/paper/1712.01727/) (2017)
- [Scaling the Scattering Transform: Deep Hybrid Networks](https://mlsanity.com/paper/1703.08961/) (2017)
- [Scattering Networks for Hybrid Representation Learning](https://mlsanity.com/paper/1809.06367/) (2018)
- [Is it enough to optimize CNN architectures on ImageNet?](https://mlsanity.com/paper/2103.09108/) (2021)
- [Harmonic Networks with Limited Training Samples](https://mlsanity.com/paper/1905.00135/) (2019)

## Built on by (most cited first, 12 of 12)

- [Towards Benchmarking and Evaluating Deepfake Detection](https://mlsanity.com/paper/2203.02115/) (2022)
- [There are no Champions in Supervised Long-Term Time Series Forecasting](https://mlsanity.com/paper/2502.14045/) (2025)
- [Which Backbone to Use: A Resource-efficient Domain Specific Comparison for Computer Vision](https://mlsanity.com/paper/2406.05612/) (2024)
- [ChimeraMix: Image Classification on Small Datasets via Masked Feature Mixing](https://mlsanity.com/paper/2202.11616/) (2022)
- [Genetic Programming-Based Evolutionary Deep Learning for Data-Efficient Image Classification](https://mlsanity.com/paper/2209.13233/) (2022)
- [A Strong Baseline for the VIPriors Data-Efficient Image Classification Challenge](https://mlsanity.com/paper/2109.13561/) (2021)
- [Infinite Class Mixup](https://mlsanity.com/paper/2305.10293/) (2023)
- [No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasets](https://mlsanity.com/paper/2309.01694/) (2023)
- [Comparing representations of biological data learned with different AI paradigms, augmenting and cropping strategies](https://mlsanity.com/paper/2203.04107/) (2022)
- [Twin: Tuning Learning Rate and Weight Decay of Deep Homogeneous Classifiers without Validation](https://mlsanity.com/paper/2403.05532/) (2024)
- [Data-Efficient Challenges in Visual Inductive Priors: A Retrospective](https://mlsanity.com/paper/2506.08612/) (2025)
- [On Hyperparameter Tuning on the Test Set](https://mlsanity.com/paper/2610.05902/) (2026)

## Abstract (arXiv)

Data-efficient image classification using deep neural networks in settings, where only small amounts of labeled data are available, has been an active research area in the recent past. However, an objective comparison between published methods is difficult, since existing works use different datasets for evaluation and often compare against untuned baselines with default hyper-parameters. We design a benchmark for data-efficient image classification consisting of six diverse datasets spanning various domains (e.g., natural images, medical imagery, satellite data) and data types (RGB, grayscale, multispectral). Using this benchmark, we re-evaluate the standard cross-entropy baseline and eight methods for data-efficient deep learning published between 2017 and 2021 at renowned venues. For a fair and realistic comparison, we carefully tune the hyper-parameters of all methods on each dataset. Surprisingly, we find that tuning learning rate, weight decay, and batch size on a separate validation split results in a highly competitive baseline, which outperforms all but one specialized method and performs competitively to the remaining one.