# Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods

arXiv: https://arxiv.org/abs/1903.11260 · HTML: https://mlsanity.com/paper/1903.11260/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2019.

## Built on by (most cited first, 30 of 62)

- [Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2101.10979/) (2021)
- [A Survey on Deep Semi-supervised Learning](https://mlsanity.com/paper/2103.00550/) (2021)
- [Image-to-Image Translation: Methods and Applications](https://mlsanity.com/paper/2101.08629/) (2021)
- [USB: A Unified Semi-supervised Learning Benchmark for Classification](https://mlsanity.com/paper/2208.07204/) (2022)
- [Expanding Small-Scale Datasets with Guided Imagination](https://mlsanity.com/paper/2211.13976/) (2022)
- [Unsupervised Point Cloud Representation Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2202.13589/) (2022)
- [Semi-Supervised and Unsupervised Deep Visual Learning: A Survey](https://mlsanity.com/paper/2208.11296/) (2022)
- [Self-supervised Contrastive Representation Learning for Semi-supervised Time-Series Classification](https://mlsanity.com/paper/2208.06616/) (2022)
- [TOV: The Original Vision Model for Optical Remote Sensing Image Understanding via Self-supervised Learning](https://mlsanity.com/paper/2204.04716/) (2022)
- [DreamDA: Generative Data Augmentation with Diffusion Models](https://mlsanity.com/paper/2403.12803/) (2024)
- [A Survey of Label-Efficient Deep Learning for 3D Point Clouds](https://mlsanity.com/paper/2305.19812/) (2023)
- [Label-Efficient Learning in Agriculture: A Comprehensive Review](https://mlsanity.com/paper/2305.14691/) (2023)
- [SEGA: Semantic Guided Attention on Visual Prototype for Few-Shot Learning](https://mlsanity.com/paper/2111.04316/) (2021)
- [Adversarial Imitation Learning with Trajectorial Augmentation and Correction](https://mlsanity.com/paper/2103.13887/) (2021)
- [Fix-A-Step: Semi-supervised Learning from Uncurated Unlabeled Data](https://mlsanity.com/paper/2208.11870/) (2022)
- [A Scoping Review of Deep Learning Methods for Photoplethysmography Data](https://mlsanity.com/paper/2401.12783/) (2024)
- [Dual-Task Mutual Learning for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2103.04708/) (2021)
- [A2Log: Attentive Augmented Log Anomaly Detection](https://mlsanity.com/paper/2109.09537/) (2021)
- [S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learning](https://mlsanity.com/paper/2309.02155/) (2023)
- [CaCo: Both Positive and Negative Samples are Directly Learnable via Cooperative-adversarial Contrastive Learning](https://mlsanity.com/paper/2203.14370/) (2022)
- [RSAM-Seg: A SAM-based Approach with Prior Knowledge Integration for Remote Sensing Image Semantic Segmentation](https://mlsanity.com/paper/2402.19004/) (2024)
- [Systematic comparison of semi-supervised and self-supervised learning for medical image classification](https://mlsanity.com/paper/2307.08919/) (2023)
- [Cutting-edge 3D Medical Image Segmentation Methods in 2020: Are Happy Families All Alike?](https://mlsanity.com/paper/2101.00232/) (2021)
- [Semi-Supervised Building Footprint Generation with Feature and Output Consistency Training](https://mlsanity.com/paper/2205.08416/) (2022)
- [Kolmogorov-Arnold Networks in Low-Data Regimes: A Comparative Study with Multilayer Perceptrons](https://mlsanity.com/paper/2409.10463/) (2024)
- [High-throughput discovery of chemical structure-polarity relationships combining automation and machine learning techniques](https://mlsanity.com/paper/2202.05962/) (2022)
- [Beyond Instance Discrimination: Relation-aware Contrastive Self-supervised Learning](https://mlsanity.com/paper/2211.01796/) (2022)
- [Semi-Supervised Object Detection: A Survey on Progress from CNN to Transformer](https://mlsanity.com/paper/2407.08460/) (2024)
- [Self-directed Machine Learning](https://mlsanity.com/paper/2201.01289/) (2022)
- [Adversarial Pretraining of Self-Supervised Deep Networks: Past, Present and Future](https://mlsanity.com/paper/2210.13463/) (2022)

## Abstract (arXiv)

Representation learning with small labeled data have emerged in many problems, since the success of deep neural networks often relies on the availability of a huge amount of labeled data that is expensive to collect. To address it, many efforts have been made on training sophisticated models with few labeled data in an unsupervised and semi-supervised fashion. In this paper, we will review the recent progresses on these two major categories of methods. A wide spectrum of models will be categorized in a big picture, where we will show how they interplay with each other to motivate explorations of new ideas. We will review the principles of learning the transformation equivariant, disentangled, self-supervised and semi-supervised representations, all of which underpin the foundation of recent progresses. Many implementations of unsupervised and semi-supervised generative models have been developed on the basis of these criteria, greatly expanding the territory of existing autoencoders, generative adversarial nets (GANs) and other deep networks by exploring the distribution of unlabeled data for more powerful representations. We will discuss emerging topics by revealing the intrinsic connections between unsupervised and semi-supervised learning, and propose in future directions to bridge the algorithmic and theoretical gap between transformation equivariance for unsupervised learning and supervised invariance for supervised learning, and unify unsupervised pretraining and supervised finetuning. We will also provide a broader outlook of future directions to unify transformation and instance equivariances for representation learning, connect unsupervised and semi-supervised augmentations, and explore the role of the self-supervised regularization for many learning problems.