Data to 7 Oct 2026

Learning Visual Representations with Caption Annotations

arXiv 2008.01392 ↗First posted 4 Aug 2020cs.CVcited by 123

Not on the map. Why?

Abstract (arXiv)

Pretraining general-purpose visual features has become a crucial part of tackling many computer vision tasks. While one can learn such features on the extensively-annotated ImageNet dataset, recent approaches have looked at ways to allow for noisy, fewer, or even no annotations to perform such pretraining. Starting from the observation that captioned images are easily crawlable, we argue that this overlooked source of information can be exploited to supervise the training of visual representations. To do so, motivated by the recent progresses in language models, we introduce em image-conditioned masked language modeling (ICMLM) -- a proxy task to learn visual representations over image-caption pairs. ICMLM consists in predicting masked words in captions by relying on visual cues. To tackle this task, we propose hybrid models, with dedicated visual and textual encoders, and we show that the visual representations learned as a by-product of solving this task transfer well to a variety of target tasks. Our experiments confirm that image captions can be leveraged to inject global and localized semantic information into visual representations. Project website: https://europe.naverlabs.com/icmlm.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 22; 2023: 38; 2024: 19; 2025: 7; 2026: 1.

123 papers.

Sort:

By area:

2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
2021Robust fine-tuning of zero-shot modelscited by 785
2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
2021Extract Free Dense Labels from CLIPcited by 580
2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315

38 more

2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
2022Prompt Distribution Learningcited by 235
2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
2021Combined Scaling for Zero-shot Transfer Learningcited by 182
2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
2023Hyperbolic Image-Text Representationscited by 132
2023Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Imagescited by 132
2023SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensingcited by 123
2021Multimodal Contrastive Training for Visual Representation Learningcited by 122
2022Task Residual for Tuning Vision-Language Modelscited by 122
2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
2023Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformerscited by 91
2023Image Captioners Are Scalable Vision Learners Toocited by 89
2021Joint Learning of Localized Representations from Medical Images and Reportscited by 81
2021CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directionscited by 78
2023An Inverse Scaling Law for CLIP Trainingcited by 76
2023Advancing Radiograph Representation Learning with Masked Record Modelingcited by 75
2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
2024POP-3D: Open-Vocabulary 3D Occupancy Prediction from Imagescited by 52
2021Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillationcited by 35
2024LocCa: Visual Pretraining with Location-aware Captionerscited by 35
2023Learning Visual Representations via Language-Guided Samplingcited by 32
2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
2023Contrastive Feature Masking Open-Vocabulary Vision Transformercited by 31
2021An Image is Worth More Than a Thousand Words: Towards Disentanglement in the Wildcited by 30
2023Multi-modal Machine Learning in Engineering Design: A Review and Future Directionscited by 27
2023CUDA: Convolution-based Unlearnable Datasetscited by 26
2023Understanding Transferable Representation Learning and Zero-shot Transfer in CLIPcited by 26
2025ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMscited by 25
2021Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examplescited by 24

50 most cited of 123 All 123 (JSON)