# Unsupervised Image Captioning

arXiv: https://arxiv.org/abs/1811.10787 · HTML: https://mlsanity.com/paper/1811.10787/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2018.

## Built on by (most cited first, 30 of 79)

- [VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning](https://mlsanity.com/paper/2102.10407/) (2021)
- [ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic](https://mlsanity.com/paper/2111.14447/) (2021)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Language Models Can See: Plugging Visual Controls in Text Generation](https://mlsanity.com/paper/2205.02655/) (2022)
- [DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training](https://mlsanity.com/paper/2303.03032/) (2023)
- [VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix](https://mlsanity.com/paper/2206.08919/) (2022)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [Relation-aware Instance Refinement for Weakly Supervised Visual Grounding](https://mlsanity.com/paper/2103.12989/) (2021)
- [Zero-shot Natural Language Video Localization](https://mlsanity.com/paper/2110.00428/) (2021)
- [Transferable Decoding with Visual Entities for Zero-Shot Image Captioning](https://mlsanity.com/paper/2307.16525/) (2023)
- [Pseudo-Q: Generating Pseudo Language Queries for Visual Grounding](https://mlsanity.com/paper/2203.08481/) (2022)
- [Multimodal Knowledge Alignment with Reinforcement Learning](https://mlsanity.com/paper/2205.12630/) (2022)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [Step-Wise Hierarchical Alignment Network for Image-Text Matching](https://mlsanity.com/paper/2106.06509/) (2021)
- [Automated Generation of Accurate & Fluent Medical X-ray Reports](https://mlsanity.com/paper/2108.12126/) (2021)
- [Exploiting Cross-Modal Prediction and Relation Consistency for Semi-Supervised Image Captioning](https://mlsanity.com/paper/2110.11767/) (2021)
- [Prompt-based Learning for Unpaired Image Captioning](https://mlsanity.com/paper/2205.13125/) (2022)
- [Language-free Training for Zero-shot Video Grounding](https://mlsanity.com/paper/2210.12977/) (2022)
- [Text-based Person Search without Parallel Image-Text Data](https://mlsanity.com/paper/2305.12964/) (2023)
- [GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths](https://mlsanity.com/paper/2408.02788/) (2024)
- [Removing Word-Level Spurious Alignment between Images and Pseudo-Captions in Unsupervised Image Captioning](https://mlsanity.com/paper/2104.13872/) (2021)
- [Aligning Source Visual and Target Language Domains for Unpaired Video Captioning](https://mlsanity.com/paper/2211.12148/) (2022)
- [Improving Cross-modal Alignment with Synthetic Pairs for Text-only Image Captioning](https://mlsanity.com/paper/2312.08865/) (2023)
- [MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning](https://mlsanity.com/paper/2308.13218/) (2023)
- [Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training](https://mlsanity.com/paper/2401.02347/) (2024)
- [Neural Attention for Image Captioning: Review of Outstanding Methods](https://mlsanity.com/paper/2111.15015/) (2021)
- [CLIP-Diffusion-LM: Apply Diffusion Model on Image Captioning](https://mlsanity.com/paper/2210.04559/) (2022)
- [Diagnosing Human-object Interaction Detectors](https://mlsanity.com/paper/2308.08529/) (2023)
- [What Makes for Good Image Captions?](https://mlsanity.com/paper/2405.00485/) (2024)

## Abstract (arXiv)

Deep neural networks have achieved great successes on the image captioning task. However, most of the existing models depend heavily on paired image-sentence datasets, which are very expensive to acquire. In this paper, we make the first attempt to train an image captioning model in an unsupervised manner. Instead of relying on manually labeled image-sentence pairs, our proposed model merely requires an image set, a sentence corpus, and an existing visual concept detector. The sentence corpus is used to teach the captioning model how to generate plausible sentences. Meanwhile, the knowledge in the visual concept detector is distilled into the captioning model to guide the model to recognize the visual concepts in an image. In order to further encourage the generated captions to be semantically consistent with the image, the image and caption are projected into a common latent space so that they can reconstruct each other. Given that the existing sentence corpora are mainly designed for linguistic research and are thus with little reference to image contents, we crawl a large-scale image description corpus of two million natural sentences to facilitate the unsupervised image captioning scenario. Experimental results show that our proposed model is able to produce quite promising results without any caption annotations.