# Can Semantic Labels Assist Self-Supervised Visual Representation Learning?

arXiv: https://arxiv.org/abs/2011.08621 · HTML: https://mlsanity.com/paper/2011.08621/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2020.

## Built on by (most cited first, 16 of 16)

- [Exploring Cross-Image Pixel Contrast for Semantic Segmentation](https://mlsanity.com/paper/2101.11939/) (2021)
- [Revisiting the Transferability of Supervised Pretraining: an MLP Perspective](https://mlsanity.com/paper/2112.00496/) (2021)
- [A Systematic Benchmarking Analysis of Transfer Learning for Medical Image Analysis](https://mlsanity.com/paper/2108.05930/) (2021)
- [Enhancing Self-supervised Video Representation Learning via Multi-level Feature Optimization](https://mlsanity.com/paper/2108.02183/) (2021)
- [Versatile Multi-Modal Pre-Training for Human-Centric Perception](https://mlsanity.com/paper/2203.13815/) (2022)
- [MaskCon: Masked Contrastive Learning for Coarse-Labelled Dataset](https://mlsanity.com/paper/2303.12756/) (2023)
- [Distance-based Hyperspherical Classification for Multi-source Open-Set Domain Adaptation](https://mlsanity.com/paper/2107.02067/) (2021)
- [Unsupervised Contrastive Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2204.08399/) (2022)
- [Decoupled Contrastive Learning for Long-Tailed Recognition](https://mlsanity.com/paper/2403.06151/) (2024)
- [Voxel-wise Cross-Volume Representation Learning for 3D Neuron Reconstruction](https://mlsanity.com/paper/2108.06522/) (2021)
- [OPERA: Omni-Supervised Representation Learning with Hierarchical Supervisions](https://mlsanity.com/paper/2210.05557/) (2022)
- [On the Effectiveness of Supervision in Asymmetric Non-Contrastive Learning](https://mlsanity.com/paper/2406.10815/) (2024)
- [Exploring the Diversity and Invariance in Yourself for Visual Pre-Training Task](https://mlsanity.com/paper/2106.00537/) (2021)
- [Fuse Local and Global Semantics in Representation Learning](https://mlsanity.com/paper/2202.13837/) (2022)
- [LAVA: Label-efficient Visual Learning and Adaptation](https://mlsanity.com/paper/2210.10317/) (2022)
- [PP-SSL : Priority-Perception Self-Supervised Learning for Fine-Grained Recognition](https://mlsanity.com/paper/2412.00134/) (2024)

## Abstract (arXiv)

Recently, contrastive learning has largely advanced the progress of unsupervised visual representation learning. Pre-trained on ImageNet, some self-supervised algorithms reported higher transfer learning performance compared to fully-supervised methods, seeming to deliver the message that human labels hardly contribute to learning transferrable visual features. In this paper, we defend the usefulness of semantic labels but point out that fully-supervised and self-supervised methods are pursuing different kinds of features. To alleviate this issue, we present a new algorithm named Supervised Contrastive Adjustment in Neighborhood (SCAN) that maximally prevents the semantic guidance from damaging the appearance feature embedding. In a series of downstream tasks, SCAN achieves superior performance compared to previous fully-supervised and self-supervised methods, and sometimes the gain is significant. More importantly, our study reveals that semantic labels are useful in assisting self-supervised methods, opening a new direction for the community.