mlsanityData to 7 Oct 2026

Text-Only Training for Image Captioning using Noise-Injected CLIP

arXiv 2211.00575 ↗First posted 1 Nov 2022cs.CV cs.AI cs.LGcited by 91

Not on the map. Why?

Abstract (arXiv)

We consider the task of image-captioning using only the CLIP model and additional text data at training time, and no additional captioned images. Our approach relies on the fact that CLIP is trained to make visual and textual embeddings similar. Therefore, we only need to learn how to translate CLIP textual embeddings back into text, and we can learn how to do this by learning a decoder for the frozen CLIP text encoder using only text. We argue that this intuition is "almost correct" because of a gap between the embedding spaces, and propose to rectify this via noise injection during training. We demonstrate the effectiveness of our approach by showing SOTA zero-shot image captioning across four benchmarks, including style transfer. Code, data, and models are available on GitHub.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 35 of its 62 references that are in the map. First posted on arXiv 1 Nov 2022.

Builds on

35 of 62 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
23 more
  1. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  2. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  3. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  4. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  5. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  6. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  7. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  8. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  9. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  10. 2016SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioningcited by 219
  11. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  12. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  13. 2019Image Captioning: Transforming Objects into Wordscited by 171
  14. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  15. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  16. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  17. 2021Dual-Level Collaborative Transformer for Image Captioningcited by 71
  18. 2019Towards Unsupervised Image Captioning with Shared Multimodal Embeddingscited by 54
  19. 2022Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioningcited by 25
  20. 2019Learning to Collocate Neural Modules for Image Captioningcited by 22
  21. 2014Learning a Recurrent Visual Representation for Image Caption Generationcited by 12
  22. 2022A Frustratingly Simple Approach for End-to-End Image Captioningcited by 12
  23. 2021Is Object Detection Necessary for Human-Object Interaction Recognition?cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 35; 2024: 27; 2025: 16; 2026: 12.

91 papers.

Sort:

By area:

  1. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  2. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  3. 2023Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lecturescited by 68
  4. 2023AutoAD: Movie Description in Contextcited by 50
  5. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  6. 2023Learned representation-guided diffusion models for large-image generationcited by 39
  7. 2023Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Modelscited by 37
  8. 2024MeaCap: Memory-Augmented Zero-shot Image Captioningcited by 35
  9. 2023Transferable Decoding with Visual Entities for Zero-Shot Image Captioningcited by 34
  10. 2023Prismer: A Vision-Language Model with Multi-Task Expertscited by 28
  11. 2023VicTR: Video-conditioned Text Representations for Activity Recognitioncited by 28
  12. 2023Language-only Efficient Training of Zero-shot Composed Image Retrievalcited by 27
38 more
  1. 2024Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Datacited by 27
  2. 2022I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervisioncited by 26
  3. 2023Auditing Gender Presentation Differences in Text-to-Image Modelscited by 22
  4. 2023Guiding Image Captioning Models Toward More Specific Captionscited by 17
  5. 2024Improving Medical Multi-modal Contrastive Learning with Expert Annotationscited by 16
  6. 2024Learning Video Context as Interleaved Multimodal Sequencescited by 16
  7. 2023Improving Cross-modal Alignment with Synthetic Pairs for Text-only Image Captioningcited by 14
  8. 2023MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioningcited by 13
  9. 2025Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generationcited by 13
  10. 2023On the Powerfulness of Textual Outlier Exposure for Visual OoD Detectioncited by 12
  11. 2024Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Trainingcited by 12
  12. 2023From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mappingcited by 11
  13. 2024As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?cited by 11
  14. 2024IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioningcited by 11
  15. 2024DistinctAD: Distinctive Audio Description Generation in Contextscited by 11
  16. 2023Beyond Generic: Enhancing Image Captioning with Real-World Knowledge using Vision-Language Pre-Training Modelcited by 10
  17. 2023CgT-GAN: CLIP-guided Text GAN for Image Captioningcited by 10
  18. 2024TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignmentcited by 10
  19. 2026Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Modelscited by 10
  20. 2023TextField3D: Towards Enhancing Open-Vocabulary 3D Generation with Noisy Text Fieldscited by 9
  21. 2024Slight Corruption in Pre-training Data Makes Better Diffusion Modelscited by 9
  22. 2023Weakly-supervised Automated Audio Captioning via text only trainingcited by 8
  23. 2023Image Captioning with Multi-Context Synthetic Datacited by 7
  24. 2023Prefix-diffusion: A Lightweight Diffusion Model for Diverse Image Captioningcited by 7
  25. 2025Scale Efficient Training for Large Datasetscited by 7
  26. 2024Learning text-to-video retrieval from image captioningcited by 6
  27. 2024Unconstrained Open Vocabulary Image Classification: Zero-Shot Transfer from Text to Image via CLIP Inversioncited by 6
  28. 2024Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learningcited by 6
  29. 2024Unleashing Text-to-Image Diffusion Prior for Zero-Shot Image Captioningcited by 6
  30. 2023ZeroGen: Zero-shot Multimodal Controllable Text Generation with Multiple Oraclescited by 5
  31. 2023ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generationcited by 4
  32. 2023Visual Captioning at Will: Describing Images and Videos Guided by a Few Stylized Sentencescited by 4
  33. 2023SymNoise: Advancing Language Model Fine-tuning with Symmetric Noisecited by 4
  34. 2024ArcSin: Adaptive ranged cosine Similarity injected noise for Language-Driven Visual Taskscited by 4
  35. 2024A Survey on Large Language Models from Concept to Implementationcited by 4
  36. 2024Towards Cross-modal Backward-compatible Representation Learning for Vision-Language Modelscited by 4
  37. 2024Text-only Synthesis for Image Captioningcited by 4
  38. 2025Multiplicity is an Inevitable and Inherent Challenge in Multimodal Learningcited by 4

50 most cited of 91 All 91 (JSON)