mlsanityData to 7 Oct 2026

Deep Learning on Small Datasets without Pre-Training using Cosine Loss

arXiv 1901.09054 ↗First posted 25 Jan 2019cs.LG cs.CV stat.MLcited by 41

Not on the map. Why?

Abstract (arXiv)

Two things seem to be indisputable in the contemporary deep learning discourse: 1. The categorical cross-entropy loss after softmax activation is the method of choice for classification. 2. Training a CNN classifier from scratch on small datasets does not work well. In contrast to this, we show that the cosine loss function provides significantly better performance than cross-entropy on datasets with only a handful of samples per class. For example, the accuracy achieved on the CUB-200-2011 dataset without pre-training is by 30% higher than with the cross-entropy loss. Further experiments on other popular datasets confirm our findings. Moreover, we demonstrate that integrating prior knowledge in the form of class hierarchies is straightforward with the cosine loss and improves classification performance further.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 13; 2023: 5; 2024: 3; 2025: 3; 2026: 3.

41 papers.

Sort:

By area:

  1. 2021A linearized framework and a new benchmark for model selection for fine-tuningcited by 41
  2. 2021Hyperbolic Busemann Learning with Ideal Prototypescited by 40
  3. 2022SR-GNN: Spatial Relation-aware Graph Neural Network for Fine-Grained Image Categorizationcited by 21
  4. 2022Integrating Language Guidance into Vision-based Deep Metric Learningcited by 19
  5. 2021Parametric Scattering Networkscited by 15
  6. 2021VIPriors 1: Visual Inductive Priors for Data-Efficient Deep Learning Challengescited by 11
  7. 2021Tune It or Don't Use It: Benchmarking Data-Efficient Image Classificationcited by 11
  8. 2021MTGLS: Multi-Task Gaze Estimation with Limited Supervisioncited by 10
  9. 2022Video-based Formative and Summative Assessment of Surgical Tasks using Deep Learningcited by 8
  10. 2022TransSleep: Transitioning-aware Attention-based Deep Neural Network for Sleep Stagingcited by 8
  11. 2022Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answerscited by 6
  12. 2022ChimeraMix: Image Classification on Small Datasets via Masked Feature Mixingcited by 4
29 more
  1. 2022Robust SAR ATR on MSTAR with Deep Learning Models trained on Full Synthetic MOCEM datacited by 4
  2. 2022Genetic Programming-Based Evolutionary Deep Learning for Data-Efficient Image Classificationcited by 4
  3. 2022Image Classification with Small Datasets: Overview and Benchmarkcited by 4
  4. 2022One-shot skill assessment in high-stakes domains with limited data via meta learningcited by 4
  5. 2023Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detectioncited by 4
  6. 2021Streaming Self-Training via Domain-Agnostic Unlabeled Imagescited by 3
  7. 2021WikiChurches: A Fine-Grained Dataset of Architectural Styles with Real-World Challengescited by 3
  8. 2022Few-Sample Traffic Prediction with Graph Networks using Locale as Relational Inductive Biasescited by 3
  9. 2021A Pilot Study For Fragment Identification Using 2D NMR and Deep Learningcited by 2
  10. 2023ProtoKD: Learning from Extremely Scarce Data for Parasite Ova Recognitioncited by 2
  11. 2024Sobolev neural network with residual weighting as a surrogate in linear and non-linear mechanicscited by 2
  12. 2024Taming CLIP for Fine-grained and Structured Visual Understanding of Museum Exhibitscited by 2
  13. 2021Object-Based Augmentation Improves Quality of Remote Sensing Semantic Segmentationcited by 1
  14. 2021Memory Wrap: a Data-Efficient and Interpretable Extension to Image Classification Modelscited by 1
  15. 2023No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasetscited by 1
  16. 2023Can Physician Judgment Enhance Model Trustworthiness? A Case Study on Predicting Pathological Lymph Nodes in Rectal Cancercited by 1
  17. 2021The Unreasonable Effectiveness of the Final Batch Normalization Layercited by 0
  18. 2021An attention-driven hierarchical multi-scale representation for visual recognitioncited by 0
  19. 2021On the Effectiveness of Neural Ensembles for Image Classification with Small Datasetscited by 0
  20. 2022Recognition of Unseen Bird Species by Learning from Field Guidescited by 0
  21. 2022'Labelling the Gaps': A Weakly Supervised Automatic Eye Gaze Estimationcited by 0
  22. 2023Class Anchor Margin Loss for Content-Based Image Retrievalcited by 0
  23. 2024The Application of Artificial Neural Network Model to Predicting the Acid Mine Drainage from Long-Term Lab Scale Kinetic Testcited by 0
  24. 2025Data-Efficient Challenges in Visual Inductive Priors: A Retrospectivecited by 0
  25. 2025Vector-based loss functions for turbulent flow field inpaintingcited by 0
  26. 2025H3Former: Hypergraph-based Semantic-Aware Aggregation via Hyperbolic Hierarchical Contrastive Loss for Fine-Grained Visual Classificationcited by 0
  27. 2026Learning Task-Invariant Properties via Dreamer: Enabling Efficient Policy Transfer for Quadruped Robotscited by 0
  28. 2026Distributional Loss for Robust Classificationcited by 0
  29. 2026Leakage-Safe Machine Learning for Hydrogen Embrittlement Detection in 316L Stainless Steel: A Region-Held-Out Evaluation of Texture and Deep Features in SEM Micrographscited by 0