mlsanityData to 7 Oct 2026

ClipCap: CLIP Prefix for Image Captioning

arXiv 2111.09734 ↗First posted 18 Nov 2021cs.CVcited by 557

Not on the map. Why?

Cited by 127 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Image captioning is a fundamental task in vision-language understanding, where the model predicts a textual informative caption to a given input image. In this paper, we present a simple approach to address this task. We use CLIP encoding as a prefix to the caption, by employing a simple mapping network, and then fine-tunes a language model to generate the image captions. The recently proposed CLIP model contains rich semantic features which were trained with textual context, making it best for vision-language perception. Our key idea is that together with a pre-trained language model (GPT2), we obtain a wide understanding of both visual and textual data. Hence, our approach only requires rather quick training to produce a competent captioning model. Without additional annotations or pre-training, it efficiently generates meaningful captions for large-scale and diverse datasets. Surprisingly, our method works well even when only the mapping network is trained, while both CLIP and the language model remain frozen, allowing a lighter architecture with less trainable parameters. Through quantitative evaluation, we demonstrate our model achieves comparable results to state-of-the-art methods on the challenging Conceptual Captions and nocaps datasets, while it is simpler, faster, and lighter. Our code is available in https://github.com/rmokady/CLIP_prefix_caption.

Key facts. Cited by 557 papers in the map (data through 7 Oct 2026). Builds on 37 of its 47 references that are in the map. First posted on arXiv 18 Nov 2021.

Builds on

37 of 47 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
25 more
  1. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  2. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  3. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  4. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  5. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  6. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  7. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  8. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  9. 2016Self-critical Sequence Training for Image Captioningcited by 544
  10. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  11. 2018nocaps: novel object captioning at scalecited by 444
  12. 2016SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioningcited by 219
  13. 2018Exploring Visual Relationship for Image Captioningcited by 215
  14. 2019Image Captioning: Transforming Objects into Wordscited by 171
  15. 2014From Captions to Visual Concepts and Backcited by 109
  16. 2021Paint by Wordcited by 107
  17. 2016Guided Open Vocabulary Image Captioning with Constrained Beam Searchcited by 105
  18. 2021Dual-Level Collaborative Transformer for Image Captioningcited by 71
  19. 2021CPTR: Full Transformer Network for Image Captioningcited by 55
  20. 2019Learning to Collocate Neural Modules for Image Captioningcited by 22
  21. 2017Skeleton Key: Image Captioning by Skeleton-Attribute Decompositioncited by 15
  22. 2018Regularizing RNNs for Caption Generation by Reconstructing The Past with The Presentcited by 15
  23. 2017Actor-Critic Sequence Training for Image Captioningcited by 13
  24. 2014Learning a Recurrent Visual Representation for Image Caption Generationcited by 12
  25. 2019Self-critical n-step Training for Image Captioningcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 80; 2023: 170; 2024: 176; 2025: 92; 2026: 37.

557 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  4. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  5. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  6. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  7. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  11. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  12. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
38 more
  1. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  2. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  3. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  4. 2023Teaching CLIP to Count to Tencited by 194
  5. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 189
  6. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  7. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  8. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  9. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  10. 2023Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrievalcited by 155
  11. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  12. 2022Linearly Mapping from Image to Text Spacecited by 139
  13. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  14. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memorycited by 122
  15. 2023VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detectioncited by 121
  16. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 118
  17. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 116
  18. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  19. 2024C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detectioncited by 111
  20. 2024A Survey on Large Language Model-Based Game Agentscited by 106
  21. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  22. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  23. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  24. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  25. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  26. 2022Text-Only Training for Image Captioning using Noise-Injected CLIPcited by 91
  27. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  28. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  29. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  30. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  31. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  32. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  33. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 79
  34. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 78
  35. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  36. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  37. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigationcited by 74
  38. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74

50 most cited of 557 All 557 (JSON)