# Deep Learning on Small Datasets without Pre-Training using Cosine Loss

arXiv: https://arxiv.org/abs/1901.09054 · HTML: https://mlsanity.com/paper/1901.09054/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jan 2019.

## Built on by (most cited first, 30 of 41)

- [A linearized framework and a new benchmark for model selection for fine-tuning](https://mlsanity.com/paper/2102.00084/) (2021)
- [Hyperbolic Busemann Learning with Ideal Prototypes](https://mlsanity.com/paper/2106.14472/) (2021)
- [SR-GNN: Spatial Relation-aware Graph Neural Network for Fine-Grained Image Categorization](https://mlsanity.com/paper/2209.02109/) (2022)
- [Integrating Language Guidance into Vision-based Deep Metric Learning](https://mlsanity.com/paper/2203.08543/) (2022)
- [Parametric Scattering Networks](https://mlsanity.com/paper/2107.09539/) (2021)
- [Tune It or Don't Use It: Benchmarking Data-Efficient Image Classification](https://mlsanity.com/paper/2108.13122/) (2021)
- [VIPriors 1: Visual Inductive Priors for Data-Efficient Deep Learning Challenges](https://mlsanity.com/paper/2103.03768/) (2021)
- [MTGLS: Multi-Task Gaze Estimation with Limited Supervision](https://mlsanity.com/paper/2110.12100/) (2021)
- [Video-based Formative and Summative Assessment of Surgical Tasks using Deep Learning](https://mlsanity.com/paper/2203.09589/) (2022)
- [TransSleep: Transitioning-aware Attention-based Deep Neural Network for Sleep Staging](https://mlsanity.com/paper/2203.12590/) (2022)
- [Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers](https://mlsanity.com/paper/2201.03425/) (2022)
- [ChimeraMix: Image Classification on Small Datasets via Masked Feature Mixing](https://mlsanity.com/paper/2202.11616/) (2022)
- [Robust SAR ATR on MSTAR with Deep Learning Models trained on Full Synthetic MOCEM data](https://mlsanity.com/paper/2206.07352/) (2022)
- [Genetic Programming-Based Evolutionary Deep Learning for Data-Efficient Image Classification](https://mlsanity.com/paper/2209.13233/) (2022)
- [Image Classification with Small Datasets: Overview and Benchmark](https://mlsanity.com/paper/2212.12478/) (2022)
- [One-shot skill assessment in high-stakes domains with limited data via meta learning](https://mlsanity.com/paper/2301.00812/) (2022)
- [Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection](https://mlsanity.com/paper/2306.00765/) (2023)
- [Streaming Self-Training via Domain-Agnostic Unlabeled Images](https://mlsanity.com/paper/2104.03309/) (2021)
- [WikiChurches: A Fine-Grained Dataset of Architectural Styles with Real-World Challenges](https://mlsanity.com/paper/2108.06959/) (2021)
- [Few-Sample Traffic Prediction with Graph Networks using Locale as Relational Inductive Biases](https://mlsanity.com/paper/2203.03965/) (2022)
- [A Pilot Study For Fragment Identification Using 2D NMR and Deep Learning](https://mlsanity.com/paper/2103.12169/) (2021)
- [ProtoKD: Learning from Extremely Scarce Data for Parasite Ova Recognition](https://mlsanity.com/paper/2309.10210/) (2023)
- [Sobolev neural network with residual weighting as a surrogate in linear and non-linear mechanics](https://mlsanity.com/paper/2407.16466/) (2024)
- [Taming CLIP for Fine-grained and Structured Visual Understanding of Museum Exhibits](https://mlsanity.com/paper/2409.01690/) (2024)
- [Object-Based Augmentation Improves Quality of Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2105.05516/) (2021)
- [Memory Wrap: a Data-Efficient and Interpretable Extension to Image Classification Models](https://mlsanity.com/paper/2106.01440/) (2021)
- [No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasets](https://mlsanity.com/paper/2309.01694/) (2023)
- [Can Physician Judgment Enhance Model Trustworthiness? A Case Study on Predicting Pathological Lymph Nodes in Rectal Cancer](https://mlsanity.com/paper/2312.09529/) (2023)
- [The Unreasonable Effectiveness of the Final Batch Normalization Layer](https://mlsanity.com/paper/2109.09016/) (2021)
- [An attention-driven hierarchical multi-scale representation for visual recognition](https://mlsanity.com/paper/2110.12178/) (2021)

## Abstract (arXiv)

Two things seem to be indisputable in the contemporary deep learning discourse: 1. The categorical cross-entropy loss after softmax activation is the method of choice for classification. 2. Training a CNN classifier from scratch on small datasets does not work well. In contrast to this, we show that the cosine loss function provides significantly better performance than cross-entropy on datasets with only a handful of samples per class. For example, the accuracy achieved on the CUB-200-2011 dataset without pre-training is by 30% higher than with the cross-entropy loss. Further experiments on other popular datasets confirm our findings. Moreover, we demonstrate that integrating prior knowledge in the form of class hierarchies is straightforward with the cosine loss and improves classification performance further.