# A Close Look at Deep Learning with Small Data

arXiv: https://arxiv.org/abs/2003.12843 · HTML: https://mlsanity.com/paper/2003.12843/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2020.

## Built on by (most cited first, 30 of 54)

- [The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning](https://mlsanity.com/paper/2304.05366/) (2023)
- [GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasets](https://mlsanity.com/paper/2204.02782/) (2022)
- [TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specialization](https://mlsanity.com/paper/2303.05506/) (2023)
- [Using DUCK-Net for Polyp Image Segmentation](https://mlsanity.com/paper/2311.02239/) (2023)
- [A Survey on Deep Tabular Learning](https://mlsanity.com/paper/2410.12034/) (2024)
- [Tune It or Don't Use It: Benchmarking Data-Efficient Image Classification](https://mlsanity.com/paper/2108.13122/) (2021)
- [Diversity vs. Recognizability: Human-like generalization in one-shot generative models](https://mlsanity.com/paper/2205.10370/) (2022)
- [This Intestine Does Not Exist: Multiscale Residual Variational Autoencoder for Realistic Wireless Capsule Endoscopy Image Generation](https://mlsanity.com/paper/2302.02150/) (2023)
- [Robots Autonomously Detecting People: A Multimodal Deep Contrastive Learning Method Robust to Intraclass Variations](https://mlsanity.com/paper/2203.00187/) (2022)
- [Data Optimization in Deep Learning: A Survey](https://mlsanity.com/paper/2310.16499/) (2023)
- [ChimeraMix: Image Classification on Small Datasets via Masked Feature Mixing](https://mlsanity.com/paper/2202.11616/) (2022)
- [Genetic Programming-Based Evolutionary Deep Learning for Data-Efficient Image Classification](https://mlsanity.com/paper/2209.13233/) (2022)
- [Image Classification with Small Datasets: Overview and Benchmark](https://mlsanity.com/paper/2212.12478/) (2022)
- [PTPI-DL-ROMs: pre-trained physics-informed deep learning-based reduced order models for nonlinear parametrized PDEs](https://mlsanity.com/paper/2405.08558/) (2024)
- [Exploiting Local Features and Range Images for Small Data Real-Time Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2410.10510/) (2024)
- [CoachMe: Decoding Sport Elements with a Reference-Based Coaching Instruction Generation Model](https://mlsanity.com/paper/2509.11698/) (2025)
- [Mind the (Data) Gap: Evaluating Vision Systems in Small Data Applications](https://mlsanity.com/paper/2504.06486/) (2025)
- [Street Review: A Participatory AI-Based Framework for Assessing Streetscape Inclusivity](https://mlsanity.com/paper/2508.11708/) (2025)
- [A Strong Baseline for the VIPriors Data-Efficient Image Classification Challenge](https://mlsanity.com/paper/2109.13561/) (2021)
- [Regularising disparity estimation via multi task learning with structured light reconstruction](https://mlsanity.com/paper/2301.08140/) (2023)
- [A new computationally efficient algorithm to solve Feature Selection for Functional Data Classification in high-dimensional spaces](https://mlsanity.com/paper/2401.05765/) (2024)
- [Transfer Learning with Informative Priors: Simple Baselines Better than Previously Reported](https://mlsanity.com/paper/2405.15583/) (2024)
- [Data Augmentation of Time-Series Data in Human Movement Biomechanics: A Scoping Review](https://mlsanity.com/paper/2504.03334/) (2025)
- [Beyond Real Data: Synthetic Data through the Lens of Regularization](https://mlsanity.com/paper/2510.08095/) (2025)
- [Robust deep learning for eye fundus images: Bridging real and synthetic data for enhancing generalization](https://mlsanity.com/paper/2203.13856/) (2022)
- [Transfer Ranking in Finance: Applications to Cross-Sectional Momentum with Data Scarcity](https://mlsanity.com/paper/2208.09968/) (2022)
- [Few-Shot Learning for Clinical Natural Language Processing Using Siamese Neural Networks](https://mlsanity.com/paper/2208.14923/) (2022)
- [Multimodal Understanding Through Correlation Maximization and Minimization](https://mlsanity.com/paper/2305.03125/) (2023)
- [No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasets](https://mlsanity.com/paper/2309.01694/) (2023)
- [Overparameterized Multiple Linear Regression as Hyper-Curve Fitting](https://mlsanity.com/paper/2404.07849/) (2024)

## Abstract (arXiv)

In this work, we perform a wide variety of experiments with different deep learning architectures on datasets of limited size. According to our study, we show that model complexity is a critical factor when only a few samples per class are available. Differently from the literature, we show that in some configurations, the state of the art can be improved using low complexity models. For instance, in problems with scarce training samples and without data augmentation, low-complexity convolutional neural networks perform comparably well or better than state-of-the-art architectures. Moreover, we show that even standard data augmentation can boost recognition performance by large margins. This result suggests the development of more complex data generation/augmentation pipelines for cases when data is limited. Finally, we show that dropout, a widely used regularization technique, maintains its role as a good regularizer even when data is scarce. Our findings are empirically validated on the sub-sampled versions of popular CIFAR-10, Fashion-MNIST and, SVHN benchmarks.