mlsanityData to 14 Oct 2026

Image Captioners Are Scalable Vision Learners Too

arXiv 2306.07915 ↗First posted 13 Jun 2023cs.CVcited by 89

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Contrastive pretraining on image-text pairs from the web is one of the most popular large-scale pretraining strategies for vision backbones, especially in the context of large multimodal models. At the same time, image captioning on this type of data is commonly considered an inferior pretraining strategy. In this paper, we perform a fair comparison of these two pretraining strategies, carefully matching training data, compute, and model capacity. Using a standard encoder-decoder transformer, we find that captioning alone is surprisingly effective: on classification tasks, captioning produces vision encoders competitive with contrastively pretrained encoders, while surpassing them on vision & language tasks. We further analyze the effect of the model architecture and scale, as well as the pretraining data on the representation quality, and find that captioning exhibits the same or better scaling behavior along these axes. Overall our results show that plain image captioning is a more powerful pretraining strategy than was previously believed.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 73 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

53 of 73 references.

References · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  12. 2021Scaling Vision Transformerscited by 999
  13. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  14. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  15. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  16. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  17. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  18. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  19. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  20. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  21. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  22. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  23. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  24. 2021OCR-free Document Understanding Transformercited by 407
  25. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  26. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  27. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  28. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  29. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  30. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  31. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  32. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  33. 2020Learning Visual Representations with Caption Annotationscited by 123
  34. 2016Learning Visual N-Grams from Web Datacited by 95
  35. 2019GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answeringcited by 78
  36. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-trainingcited by 67
  37. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
  38. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 55

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 32; 2025: 29; 2026: 17.

89 papers.

Sort:

By area:

  1. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  2. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  3. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  4. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  5. 2024The pitfalls of next-token predictioncited by 160
  6. 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
  7. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 126
  8. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  9. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 109
  10. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 97
  11. 2023GIVT: Generative Infinite-Vocabulary Transformerscited by 80
  12. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 80
38 more
  1. 2023Foundational Models in Medical Imaging: A Comprehensive Survey and Future Visioncited by 76
  2. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
  3. 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
  4. 2025Scaling Language-Free Visual Representation Learningcited by 66
  5. 2023Intriguing properties of generative classifierscited by 52
  6. 2023SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformerscited by 49
  7. 2025SensorLM: Learning the Language of Wearable Sensorscited by 49
  8. 2024JetFormer: An Autoregressive Generative Model of Raw Images and Textcited by 46
  9. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 42
  10. 2025A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1cited by 37
  11. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 37
  12. 2024LocCa: Visual Pretraining with Location-aware Captionerscited by 35
  13. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  14. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  15. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformercited by 30
  16. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 29
  17. 2025Can Masked Autoencoders Also Listen to Birds?cited by 26
  18. 2025ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMscited by 25
  19. 2024The Hard Positive Truth about Vision-Language Compositionalitycited by 22
  20. 2025Comprehensive language-image pre-training for 3D medical image understandingcited by 22
  21. 2025E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-trainingcited by 22
  22. 2024ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMscited by 21
  23. 2024Nomic Embed Vision: Expanding the Latent Spacecited by 19
  24. 2025Scaling Pre-training to One Hundred Billion Data for Vision Language Modelscited by 19
  25. 2024VLRM: Vision-Language Models act as Reward Models for Image Captioningcited by 18
  26. 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 18
  27. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Modelscited by 16
  28. 2025Beyond Multi-Token Prediction: Pretraining LLMs with Future Summariescited by 15
  29. 2024Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmerscited by 14
  30. 2024Demonstrating and Reducing Shortcuts in Vision-Language Representation Learningcited by 14
  31. 2024Locality Alignment Improves Vision-Language Modelscited by 13
  32. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 13
  33. 2026TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignmentcited by 13
  34. 2024CLoVe: Encoding Compositional Language in Contrastive Vision-Language Modelscited by 12
  35. 2024Hyperbolic Learning with Synthetic Captions for Open-World Detectioncited by 12
  36. 2026SleepLM: Natural-Language Intelligence for Human Sleepcited by 12
  37. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encodercited by 11
  38. 2024Classification Done Right for Vision-Language Pre-Trainingcited by 10

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language models
  2. 2026How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining
  3. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretraining
  4. 2021Contrastive Vision-Language Pre-training with Limited Resources
  5. 2024DreamLIP: Language-Image Pre-training with Long Captions
  6. 2025Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from Translation
  7. 2023CapsFusion: Rethinking Image-Text Data at Scale
  8. 2022A Survey of Vision-Language Pre-Trained Models