mlsanityData to 7 Oct 2026

CoCa: Contrastive Captioners are Image-Text Foundation Models

arXiv 2205.01917 ↗First posted 4 May 2022cs.CV cs.LG cs.MMcited by 1,265

Not on the map. Why?

Cited by 377 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Exploring large-scale pretrained foundation models is of significant interest in computer vision because these models can be quickly transferred to many downstream tasks. This paper presents Contrastive Captioner (CoCa), a minimalist design to pretrain an image-text encoder-decoder foundation model jointly with contrastive loss and captioning loss, thereby subsuming model capabilities from contrastive approaches like CLIP and generative methods like SimVLM. In contrast to standard encoder-decoder transformers where all decoder layers attend to encoder outputs, CoCa omits cross-attention in the first half of decoder layers to encode unimodal text representations, and cascades the remaining decoder layers which cross-attend to the image encoder for multimodal image-text representations. We apply a contrastive loss between unimodal image and text embeddings, in addition to a captioning loss on the multimodal decoder outputs which predicts text tokens autoregressively. By sharing the same computational graph, the two training objectives are computed efficiently with minimal overhead. CoCa is pretrained end-to-end and from scratch on both web-scale alt-text data and annotated images by treating all labels simply as text, seamlessly unifying natural language supervision for representation learning. Empirically, CoCa achieves state-of-the-art performance with zero-shot transfer or minimal task-specific adaptation on a broad range of downstream tasks, spanning visual recognition (ImageNet, Kinetics-400/600/700, Moments-in-Time), crossmodal retrieval (MSCOCO, Flickr30K, MSR-VTT), multimodal understanding (VQA, SNLI-VE, NLVR2), and image captioning (MSCOCO, NoCaps). Notably on ImageNet classification, CoCa obtains 86.3% zero-shot top-1 accuracy, 90.6% with a frozen encoder and learned classification head, and new state-of-the-art 91.0% top-1 accuracy on ImageNet with a finetuned encoder.

Key facts. Cited by 1,265 papers in the map (data through 7 Oct 2026). Builds on 73 of its 88 references that are in the map. First posted on arXiv 4 May 2022.

Builds on

73 of 88 references.

References · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  6. 2017The Kinetics Human Action Video Datasetcited by 1,894
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2021ViViT: A Video Vision Transformercited by 1,610
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  13. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  14. 2019Natural Adversarial Examplescited by 1,401
  15. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  16. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  17. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  18. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  19. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  20. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  21. 2021Scaling Vision Transformerscited by 993
  22. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  23. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  24. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  25. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  26. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  27. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  28. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  29. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  30. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  31. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  32. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  33. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  34. 2016Self-critical Sequence Training for Image Captioningcited by 544
  35. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  36. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  37. 2018nocaps: novel object captioning at scalecited by 444
  38. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 136; 2023: 412; 2024: 332; 2025: 266; 2026: 119.

1,265 papers.

Sort:

By area:

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  10. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  11. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  12. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
38 more
  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  3. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  4. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  5. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  6. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  7. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  8. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  9. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  10. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  11. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 489
  12. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  13. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  14. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  15. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  16. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  17. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  18. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  19. 2022Multimodal Learning with Transformers: A Surveycited by 399
  20. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  21. 2025Kimi-VL Technical Reportcited by 381
  22. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  23. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  24. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  25. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  26. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  27. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  28. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  29. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  30. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  31. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  32. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  33. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  34. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  35. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  36. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  37. 2023Generating Images with Multimodal Language Modelscited by 259
  38. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245

50 most cited of 1,265 All 1,265 (JSON)