mlsanityData to 14 Oct 2026

ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic

arXiv 2111.14447 ↗First posted 29 Nov 2021cs.CV cs.AI cs.CLcited by 150

Not on the map. Why?

Cited by 21 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent text-to-image matching models apply contrastive learning to large corpora of uncurated pairs of images and sentences. While such models can provide a powerful score for matching and subsequent zero-shot tasks, they are not capable of generating caption given an image. In this work, we repurpose such models to generate a descriptive text given an image at inference time, without any further training or tuning steps. This is done by combining the visual-semantic model with a large language model, benefiting from the knowledge in both web-scale models. The resulting captions are much less restrictive than those obtained by supervised captioning methods. Moreover, as a zero-shot learning method, it is extremely flexible and we demonstrate its ability to perform image arithmetic in which the inputs can be either images or text, and the output is a sentence. This enables novel high-level vision capabilities such as comparing two images or solving visual analogy tests. Our code is available at: https://github.com/YoadTew/zero-shot-image-to-text.

Key facts. Cited by 150 papers in the map (data through 14 Oct 2026). Builds on 66 of its 80 references that are in the map. First posted on arXiv 29 Nov 2021.

Builds on

66 of 80 references.

References · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  9. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  10. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
38 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  4. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  5. 2014How transferable are features in deep neural networks?cited by 1,225
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  7. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  8. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  9. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  10. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  11. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  12. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  13. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 484
  14. 2018nocaps: novel object captioning at scalecited by 444
  15. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 376
  16. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  17. 2016Visual Relationship Detection with Language Priorscited by 334
  18. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  19. 2017Toward Controlled Generation of Textcited by 272
  20. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 255
  21. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  22. 2017Style Transfer from Non-Parallel Text by Cross-Alignmentcited by 248
  23. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 232
  24. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  25. 2018Exploring Visual Relationship for Image Captioningcited by 215
  26. 2020X-Linear Attention Networks for Image Captioningcited by 198
  27. 2019Image Captioning: Transforming Objects into Wordscited by 171
  28. 2018Neural Baby Talkcited by 107
  29. 2016Guided Open Vocabulary Image Captioning with Constrained Beam Searchcited by 105
  30. 2016Learning to Translate in Real-time with Neural Machine Translationcited by 97
  31. 2014Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)cited by 94
  32. 2018Unsupervised Image Captioningcited by 79
  33. 2021Dual-Level Collaborative Transformer for Image Captioningcited by 71
  34. 2020Normalized and Geometry-Aware Self-Attention Network for Image Captioningcited by 63
  35. 2020Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropiescited by 57
  36. 2019Towards Unsupervised Image Captioning with Shared Multimodal Embeddingscited by 54
  37. 2017Convolutional Image Captioningcited by 52
  38. 2019A Simple Baseline for Audio-Visual Scene-Aware Dialogcited by 52

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 22; 2023: 53; 2024: 43; 2025: 21; 2026: 11.

150 papers.

Sort:

By area:

  1. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  2. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  3. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  4. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 363
  5. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 315
  6. 2023Teaching CLIP to Count to Tencited by 198
  7. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  8. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  9. 2022Language Models are General-Purpose Interfacescited by 97
  10. 2022Text-Only Training for Image Captioning using Noise-Injected CLIPcited by 91
  11. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discoverycited by 91
  12. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 90
38 more
  1. 2022Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?cited by 90
  2. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  3. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  4. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  5. 2023AutoSAM: Adapting SAM to Medical Images by Overloading the Prompt Encodercited by 70
  6. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 67
  7. 2023Text-To-Concept (and Back) via Cross-Model Alignmentcited by 64
  8. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Modelscited by 61
  9. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  10. 2023BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decodingcited by 36
  11. 2024A Vision Check-up for Language Modelscited by 36
  12. 2022Toxicity Detection with Generative Prompt-based Inferencecited by 35
  13. 2024MeaCap: Memory-Augmented Zero-shot Image Captioningcited by 35
  14. 2023Transferable Decoding with Visual Entities for Zero-Shot Image Captioningcited by 34
  15. 2023ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishingcited by 33
  16. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 32
  17. 2022Multimodal Knowledge Alignment with Reinforcement Learningcited by 30
  18. 2022Composing Ensembles of Pre-trained Models via Iterative Consensuscited by 28
  19. 2023Prismer: A Vision-Language Model with Multi-Task Expertscited by 28
  20. 2023Label-Free Event-based Object Recognition via Joint Learning with Image Reconstruction from Eventscited by 28
  21. 2024Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Datacited by 27
  22. 2022I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervisioncited by 26
  23. 2022No Token Left Behind: Explainability-Aided Image Classification and Generationcited by 25
  24. 2022Zero-Shot Video Captioning with Evolving Pseudo-Tokenscited by 25
  25. 2022Noise-aware Learning from Web-crawled Image-Text Data for Image Captioningcited by 22
  26. 2023AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generationcited by 22
  27. 2023ExFaceGAN: Exploring Identity Directions in GAN's Learned Latent Space for Synthetic Identity Generationcited by 22
  28. 2023Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learningcited by 20
  29. 2023AvatarFusion: Zero-shot Generation of Clothing-Decoupled 3D Avatars Using 2D Diffusioncited by 20
  30. 2023BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivitycited by 19
  31. 2024Emergent Visual-Semantic Hierarchies in Image-Text Representationscited by 19
  32. 2024SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIPcited by 19
  33. 2023Zero-guidance Segmentation Using Zero Segment Labelscited by 18
  34. 2024T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretrainingcited by 18
  35. 2023LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Promptingcited by 17
  36. 2023Guiding Image Captioning Models Toward More Specific Captionscited by 17
  37. 2022Large Language Models are Pretty Good Zero-Shot Video Game Bug Detectorscited by 14
  38. 2023MAGVLT: Masked Generative Vision-and-Language Transformercited by 14

50 most cited of 150 All 150 (JSON)