mlsanityData to 14 Oct 2026

Self-supervised learning of visual features through embedding images into text topic spaces

arXiv 1705.08631 ↗First posted 24 May 2017cs.CVcited by 43

Not on the map. Why?

Abstract (arXiv)

End-to-end training from scratch of current deep architectures for new computer vision problems would require Imagenet-scale datasets, and this is not always possible. In this paper we present a method that is able to take advantage of freely available multi-modal content to train computer vision algorithms without human supervision. We put forward the idea of performing self-supervised learning of visual features by mining a large scale corpus of multi-modal (text and image) documents. We show that discriminative visual features can be learnt efficiently by training a CNN to predict the semantic context in which a particular image is more probable to appear as an illustration. For this we leverage the hidden semantic structures discovered in the text corpus with a well-known topic modeling technique. Our experiments demonstrate state of the art performance in image classification, object detection, and multi-modal retrieval compared to recent self-supervised or natural-supervised approaches.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 9; 2023: 14; 2024: 9; 2025: 1; 2026: 1.

43 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  3. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  4. 2021Extract Free Dense Labels from CLIPcited by 583
  5. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  6. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  7. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  8. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 124
  9. 2022Task Residual for Tuning Vision-Language Modelscited by 123
  10. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  11. 2022P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervisioncited by 52
  12. 2022Multitask Vision-Language Prompt Tuningcited by 51
31 more
  1. 2022The Challenges of Continuous Self-Supervised Learningcited by 36
  2. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  3. 2024Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrievalcited by 36
  4. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  5. 2023Vocabulary-free Image Classificationcited by 35
  6. 2023Understanding Transferable Representation Learning and Zero-shot Transfer in CLIPcited by 26
  7. 2023Domain-Controlled Prompt Learningcited by 22
  8. 2023Learning to Adapt CLIP for Few-Shot Monocular Depth Estimationcited by 20
  9. 2023Seeing in Flowing: Adapting CLIP for Action Recognition with Motion Prompts Learningcited by 16
  10. 2021Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matchingcited by 13
  11. 2023Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Modelscited by 13
  12. 2021Exploring Visual Engagement Signals for Representation Learningcited by 12
  13. 2022iCAR: Bridging Image Classification and Image-text Alignment for Visual Recognitioncited by 12
  14. 2023Cross-Modal Concept Learning and Inference for Vision-Language Modelscited by 12
  15. 2023CoPL: Contextual Prompt Learning for Vision-Language Understandingcited by 10
  16. 2024Vocabulary-free Image Classification and Semantic Segmentationcited by 8
  17. 2023RaSP: Relation-aware Semantic Prior for Weakly Supervised Incremental Segmentationcited by 6
  18. 2024DeDe: Detecting Backdoor Samples for SSL Encoders via Decoderscited by 6
  19. 2022FeaRLESS: Feature Refinement Loss for Ensembling Self-Supervised Learning Features in Robust End-to-end Speech Recognitioncited by 5
  20. 2024Benchmarking Zero-Shot Robustness of Multimodal Foundation Models: A Pilot Studycited by 5
  21. 2023Zero-shot Building Attribute Extraction from Large-Scale Vision and Language Modelscited by 4
  22. 2022Bridging Speech and Textual Pre-trained Models with Unsupervised ASRcited by 3
  23. 2023Text-Only Training for Visual Storytellingcited by 3
  24. 2023SimCon Loss with Multiple Views for Text Supervised Semantic Segmentationcited by 2
  25. 2024A Multi-Modal Deep Learning Based Approach for House Price Predictioncited by 2
  26. 2021RRLFSOR: An Efficient Self-Supervised Learning Strategy of Graph Convolutional Networkscited by 0
  27. 2024The Role of Generative Systems in Historical Photography Management: A Case Study on Catalan Archivescited by 0
  28. 2024CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidancecited by 0
  29. 2024Scalable Early Childhood Reading Performance Predictioncited by 0
  30. 2025AmPLe: Supporting Vision-Language Models via Adaptive-Debiased Ensemble Multi-Prompt Learningcited by 0
  31. 2026Visual Model Checking: Graph-Based Inference of Visual Routines for Image Retrievalcited by 0