# Multi-Label Self-Supervised Learning with Scene Images

arXiv: https://arxiv.org/abs/2308.03286 · HTML: https://mlsanity.com/paper/2308.03286/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 59 references that are in the map.
- First posted on arXiv 7 Aug 2023.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2104.14548/) (2021)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2011.10043/) (2020)
- [DetCo: Unsupervised Contrastive Learning for Object Detection](https://mlsanity.com/paper/2102.04803/) (2021)
- [Self-supervised Pretraining of Visual Features in the Wild](https://mlsanity.com/paper/2103.01988/) (2021)
- [Efficient Visual Pretraining with Contrastive Detection](https://mlsanity.com/paper/2103.10957/) (2021)
- [Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition](https://mlsanity.com/paper/1908.07325/) (2019)

## Built on by (most cited first, 17 of 17)

- [Self-Supervised Visual Preference Alignment](https://mlsanity.com/paper/2404.10501/) (2024)
- [Multi-Label Guided Soft Contrastive Learning for Efficient Earth Observation Pretraining](https://mlsanity.com/paper/2405.20462/) (2024)
- [DTL: Disentangled Transfer Learning for Visual Recognition](https://mlsanity.com/paper/2312.07856/) (2023)
- [Deep Learning for Multi-Label Learning: A Comprehensive Survey](https://mlsanity.com/paper/2401.16549/) (2024)
- [Multi-label Cluster Discrimination for Visual Representation Learning](https://mlsanity.com/paper/2407.17331/) (2024)
- [Rectify the Regression Bias in Long-Tailed Object Detection](https://mlsanity.com/paper/2401.15885/) (2024)
- [Continual SFT Matches Multimodal RLHF with Negative Supervision](https://mlsanity.com/paper/2411.14797/) (2024)
- [DiffuLT: How to Make Diffusion Model Useful for Long-tail Recognition](https://mlsanity.com/paper/2403.05170/) (2024)
- [Multi-Token Enhancing for Vision Representation Learning](https://mlsanity.com/paper/2411.15787/) (2024)
- [Enhancing Representations through Heterogeneous Self-Supervised Learning](https://mlsanity.com/paper/2310.05108/) (2023)
- [SHAPE : Self-Improved Visual Preference Alignment by Iteratively Generating Holistic Winner](https://mlsanity.com/paper/2503.04858/) (2025)
- [Multi-label out-of-distribution detection via evidential learning](https://mlsanity.com/paper/2502.18224/) (2025)
- [Cross-Modal Consistency Learning for Sign Language Recognition](https://mlsanity.com/paper/2503.12485/) (2025)
- [CoST: Semantic-Aware Urban Understanding via Spatial-Temporal Alignment](https://mlsanity.com/paper/2608.21041/) (2026)
- [Token-Level Inference-Time Alignment for Vision-Language Models](https://mlsanity.com/paper/2510.21794/) (2025)
- [Rethinking Multi-Label Image Classification With Deep Learning: Taxonomy, Challenge, and Outlook](https://mlsanity.com/paper/2607.00839/) (2026)
- [Adapting Dense Vision-Language Relationships for Multi-label Classification with Partial Label](https://mlsanity.com/paper/2608.22313/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) methods targeting scene images have seen a rapid growth recently, and they mostly rely on either a dedicated dense matching mechanism or a costly unsupervised object discovery module. This paper shows that instead of hinging on these strenuous operations, quality image representations can be learned by treating scene/multi-label image SSL simply as a multi-label classification problem, which greatly simplifies the learning framework. Specifically, multiple binary pseudo-labels are assigned for each input image by comparing its embeddings with those in two dictionaries, and the network is optimized using the binary cross entropy loss. The proposed method is named Multi-Label Self-supervised learning (MLS). Visualizations qualitatively show that clearly the pseudo-labels by MLS can automatically find semantically similar pseudo-positive pairs across different images to facilitate contrastive learning. MLS learns high quality representations on MS-COCO and achieves state-of-the-art results on classification, detection and segmentation benchmarks. At the same time, MLS is much simpler than existing methods, making it easier to deploy and for further exploration.