mlsanityData to 7 Oct 2026

Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning

arXiv 2207.07635 ↗First posted 15 Jul 2022cs.CV cs.LG stat.MLcited by 39

Not on the map. Why?

Abstract (arXiv)

The development of CLIP [Radford et al., 2021] has sparked a debate on whether language supervision can result in vision models with more transferable representations than traditional image-only methods. Our work studies this question through a carefully controlled comparison of two approaches in terms of their ability to learn representations that generalize to downstream classification tasks. We find that when the pre-training dataset meets certain criteria -- it is sufficiently large and contains descriptive captions with low variability -- image-only methods do not match CLIP's transfer performance, even when they are trained with more image data. However, contrary to what one might expect, there are practical settings in which these criteria are not met, wherein added supervision through captions is actually detrimental. Motivated by our findings, we devise simple prescriptions to enable CLIP to better leverage the language information present in existing pre-training datasets.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 52 of its 76 references that are in the map. First posted on arXiv 15 Jul 2022.

Builds on

52 of 76 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2020Exploring Simple Siamese Representation Learningcited by 2,853
38 more
  1. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  2. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  3. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  5. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  7. 2019Natural Adversarial Examplescited by 1,401
  8. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  9. 2014How transferable are features in deep neural networks?cited by 1,216
  10. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  11. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  12. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  13. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  14. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  15. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  16. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  17. 2018Do Better ImageNet Models Transfer Better?cited by 560
  18. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  19. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  20. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  21. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  22. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  23. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  24. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  25. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  26. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  27. 2020Representation Learning via Invariant Causal Mechanismscited by 197
  28. 2016What makes ImageNet good for transfer learning?cited by 192
  29. 2020Learning Robust Representations via Multi-View Information Bottleneckcited by 189
  30. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  31. 2014Return of the Devil in the Details: Delving Deep into Convolutional Netscited by 122
  32. 2021When Does Contrastive Visual Representation Learning Work?cited by 92
  33. 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
  34. 2020On Mutual Information in Contrastive Learning for Visual Representationscited by 54
  35. 2020A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluationcited by 52
  36. 2021Lossy Compression for Lossless Predictioncited by 51
  37. 2021Optimal Representations for Covariate Shiftcited by 51
  38. 2014Analyzing the Performance of Multilayer Neural Networks for Object Recognitioncited by 29

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 25; 2024: 6; 2025: 1; 2026: 1.

39 papers.

Sort:

By area:

  1. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  2. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  3. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  4. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  5. 2024An Introduction to Vision-Language Modelingcited by 175
  6. 2023Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Modelscited by 171
  7. 2022DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Modelscited by 167
  8. 2023BioCLIP: A Vision Foundation Model for the Tree of Lifecited by 159
  9. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  10. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  11. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  12. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
27 more
  1. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  2. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  3. 2023Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity?cited by 35
  4. 2024Effective pruning of web-scale datasets based on complexity of concept clusterscited by 34
  5. 2023On the Generalization of Multi-modal Contrastive Learningcited by 30
  6. 2023CiT: Curation in Training for Effective Vision-Language Datacited by 27
  7. 2023MLLMs-Augmented Visual-Language Representation Learningcited by 21
  8. 2023The Devil is in the Details: A Deep Dive into the Rabbit Hole of Data Filteringcited by 19
  9. 2023The Role of Pre-training Data in Transfer Learningcited by 17
  10. 2023Effective Robustness against Natural Distribution Shifts for Models with Different Training Datacited by 13
  11. 2023Few-shot Action Recognition with Captioning Foundation Modelscited by 9
  12. 2023Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute associationcited by 7
  13. 2022Learning Transferable Spatiotemporal Representations from Natural Script Knowledgecited by 6
  14. 2023Evaluating Self-Supervised Learning via Risk Decompositioncited by 6
  15. 2023Large Language Models can Share Images, Too!cited by 5
  16. 2024Web-Scale Visual Entity Recognition: An LLM-Driven Data Approachcited by 5
  17. 2022Caption supervision enables robust learnerscited by 4
  18. 2023Understanding the Robustness of Multi-modal Contrastive Learning to Distribution Shiftcited by 4
  19. 2023CapText: Large Language Model-based Caption Generation From Image Context and Descriptioncited by 3
  20. 2023Distributionally Robust Classification on a Data Budgetcited by 2
  21. 2023Exploring Dataset-Scale Indicators of Data Qualitycited by 1
  22. 2023Boosting Audio-visual Zero-shot Learning with Large Language Modelscited by 1
  23. 2024Language Plays a Pivotal Role in the Object-Attribute Compositional Generalization of CLIPcited by 1
  24. 2024ACE: Action Concept Enhancement of Video-Language Models in Procedural Videoscited by 1
  25. 2026ReCap: Lightweight Referential Grounding for Coherent Story Visualizationcited by 1
  26. 2023Targeted Image Data Augmentation Increases Basic Skills Captioning Robustnesscited by 0
  27. 2025Mining Contextualized Visual Associations from Images for Creativity Understandingcited by 0