# An Analysis of Unsupervised Pre-training in Light of Recent Advances

arXiv: https://arxiv.org/abs/1412.6597 · HTML: https://mlsanity.com/paper/1412.6597/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Dec 2014.

## Built on by (most cited first, 8 of 8)

- [Self-Paced Contrastive Learning for Semi-supervised Medical Image Segmentation with Meta-labels](https://mlsanity.com/paper/2107.13741/) (2021)
- [Deep transfer learning for image classification: a survey](https://mlsanity.com/paper/2205.09904/) (2022)
- [Understanding Optimal Feature Transfer via a Fine-Grained Bias-Variance Analysis](https://mlsanity.com/paper/2404.12481/) (2024)
- [OscNet: Machine Learning on CMOS Oscillator Networks](https://mlsanity.com/paper/2502.07192/) (2025)
- [Towards Contrastive Learning in Music Video Domain](https://mlsanity.com/paper/2309.00347/) (2023)
- [Domain Adaptation for Facial Expression Classifier via Domain Discrimination and Gradient Reversal](https://mlsanity.com/paper/2106.01467/) (2021)
- [Generative Transfer Learning: Covid-19 Classification with a few Chest X-ray Images](https://mlsanity.com/paper/2208.05305/) (2022)
- [Examining the Effect of Pre-training on Time Series Classification](https://mlsanity.com/paper/2309.05256/) (2023)

## Abstract (arXiv)

Convolutional neural networks perform well on object recognition because of a number of recent advances: rectified linear units (ReLUs), data augmentation, dropout, and large labelled datasets. Unsupervised data has been proposed as another way to improve performance. Unfortunately, unsupervised pre-training is not used by state-of-the-art methods leading to the following question: Is unsupervised pre-training still useful given recent advances? If so, when? We answer this in three parts: we 1) develop an unsupervised method that incorporates ReLUs and recent unsupervised regularization techniques, 2) analyze the benefits of unsupervised pre-training compared to data augmentation and dropout on CIFAR-10 while varying the ratio of unsupervised to supervised samples, 3) verify our findings on STL-10. We discover unsupervised pre-training, as expected, helps when the ratio of unsupervised to supervised samples is high, and surprisingly, hurts when the ratio is low. We also use unsupervised pre-training with additional color augmentation to achieve near state-of-the-art performance on STL-10.