# Self-supervised learning of visual features through embedding images into text topic spaces

arXiv: https://arxiv.org/abs/1705.08631 · HTML: https://mlsanity.com/paper/1705.08631/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2017.

## Built on by (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Multi-View Transformer for 3D Visual Grounding](https://mlsanity.com/paper/2204.02174/) (2022)
- [Multimodal Contrastive Training for Visual Representation Learning](https://mlsanity.com/paper/2104.12836/) (2021)
- [Task Residual for Tuning Vision-Language Models](https://mlsanity.com/paper/2211.10277/) (2022)
- [A Comprehensive Survey of Foundation Models in Medicine](https://mlsanity.com/paper/2406.10729/) (2024)
- [P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervision](https://mlsanity.com/paper/2205.02300/) (2022)
- [Multitask Vision-Language Prompt Tuning](https://mlsanity.com/paper/2211.11720/) (2022)
- [The Challenges of Continuous Self-Supervised Learning](https://mlsanity.com/paper/2203.12710/) (2022)
- [Retrieval-Enhanced Contrastive Vision-Text Models](https://mlsanity.com/paper/2306.07196/) (2023)
- [Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2403.16005/) (2024)
- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [Vocabulary-free Image Classification](https://mlsanity.com/paper/2306.00917/) (2023)
- [Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP](https://mlsanity.com/paper/2310.00927/) (2023)
- [Domain-Controlled Prompt Learning](https://mlsanity.com/paper/2310.07730/) (2023)
- [Learning to Adapt CLIP for Few-Shot Monocular Depth Estimation](https://mlsanity.com/paper/2311.01034/) (2023)
- [Seeing in Flowing: Adapting CLIP for Action Recognition with Motion Prompts Learning](https://mlsanity.com/paper/2308.04828/) (2023)
- [Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matching](https://mlsanity.com/paper/2110.02623/) (2021)
- [Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models](https://mlsanity.com/paper/2311.17091/) (2023)
- [Exploring Visual Engagement Signals for Representation Learning](https://mlsanity.com/paper/2104.07767/) (2021)
- [iCAR: Bridging Image Classification and Image-text Alignment for Visual Recognition](https://mlsanity.com/paper/2204.10760/) (2022)
- [Cross-Modal Concept Learning and Inference for Vision-Language Models](https://mlsanity.com/paper/2307.15460/) (2023)
- [CoPL: Contextual Prompt Learning for Vision-Language Understanding](https://mlsanity.com/paper/2307.00910/) (2023)
- [Vocabulary-free Image Classification and Semantic Segmentation](https://mlsanity.com/paper/2404.10864/) (2024)
- [RaSP: Relation-aware Semantic Prior for Weakly Supervised Incremental Segmentation](https://mlsanity.com/paper/2305.19879/) (2023)
- [DeDe: Detecting Backdoor Samples for SSL Encoders via Decoders](https://mlsanity.com/paper/2411.16154/) (2024)

## Abstract (arXiv)

End-to-end training from scratch of current deep architectures for new computer vision problems would require Imagenet-scale datasets, and this is not always possible. In this paper we present a method that is able to take advantage of freely available multi-modal content to train computer vision algorithms without human supervision. We put forward the idea of performing self-supervised learning of visual features by mining a large scale corpus of multi-modal (text and image) documents. We show that discriminative visual features can be learnt efficiently by training a CNN to predict the semantic context in which a particular image is more probable to appear as an illustration. For this we leverage the hidden semantic structures discovered in the text corpus with a well-known topic modeling technique. Our experiments demonstrate state of the art performance in image classification, object detection, and multi-modal retrieval compared to recent self-supervised or natural-supervised approaches.