# Self-supervised Label Augmentation via Input Transformations

arXiv: https://arxiv.org/abs/1910.05872 · HTML: https://mlsanity.com/paper/1910.05872/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2019.

## Built on by (most cited first, 15 of 15)

- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Data Augmentation techniques in time series domain: A survey and taxonomy](https://mlsanity.com/paper/2206.13508/) (2022)
- [Distilling a Powerful Student Model via Online Knowledge Distillation](https://mlsanity.com/paper/2103.14473/) (2021)
- [Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity](https://mlsanity.com/paper/2111.02338/) (2021)
- [A Review of Predictive and Contrastive Self-supervised Learning for Medical Images](https://mlsanity.com/paper/2302.05043/) (2023)
- [Learning From Biased Soft Labels](https://mlsanity.com/paper/2302.08155/) (2023)
- [Universal Adaptive Data Augmentation](https://mlsanity.com/paper/2207.06658/) (2022)
- [Understanding the Role of Self-Supervised Learning in Out-of-Distribution Detection Task](https://mlsanity.com/paper/2110.13435/) (2021)
- [Frameless Graph Knowledge Distillation](https://mlsanity.com/paper/2307.06631/) (2023)
- [ATAS: Any-to-Any Self-Distillation for Enhanced Open-Vocabulary Dense Prediction](https://mlsanity.com/paper/2506.08678/) (2025)
- [A Multi-Stage Attentive Transfer Learning Framework for Improving COVID-19 Diagnosis](https://mlsanity.com/paper/2101.05410/) (2021)
- [Dynamic Rectification Knowledge Distillation](https://mlsanity.com/paper/2201.11319/) (2022)
- [Right Time to Learn:Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation](https://mlsanity.com/paper/2502.06192/) (2025)
- [RTF-Q: Efficient Unsupervised Domain Adaptation with Retraining-free Quantization](https://mlsanity.com/paper/2408.05752/) (2024)
- [Enriching Knowledge Distillation with Intra-Class Contrastive Learning](https://mlsanity.com/paper/2509.22053/) (2025)

## Abstract (arXiv)

Self-supervised learning, which learns by constructing artificial labels given only the input signals, has recently gained considerable attention for learning representations with unlabeled datasets, i.e., learning without any human-annotated supervision. In this paper, we show that such a technique can be used to significantly improve the model accuracy even under fully-labeled datasets. Our scheme trains the model to learn both original and self-supervised tasks, but is different from conventional multi-task learning frameworks that optimize the summation of their corresponding losses. Our main idea is to learn a single unified task with respect to the joint distribution of the original and self-supervised labels, i.e., we augment original labels via self-supervision of input transformation. This simple, yet effective approach allows to train models easier by relaxing a certain invariant constraint during learning the original and self-supervised tasks simultaneously. It also enables an aggregated inference which combines the predictions from different augmentations to improve the prediction accuracy. Furthermore, we propose a novel knowledge transfer technique, which we refer to as self-distillation, that has the effect of the aggregated inference in a single (faster) inference. We demonstrate the large accuracy improvement and wide applicability of our framework on various fully-supervised settings, e.g., the few-shot and imbalanced classification scenarios.