# Removing Word-Level Spurious Alignment between Images and Pseudo-Captions in Unsupervised Image Captioning

arXiv: https://arxiv.org/abs/2104.13872 · HTML: https://mlsanity.com/paper/2104.13872/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 28 Apr 2021.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [nocaps: novel object captioning at scale](https://mlsanity.com/paper/1812.08658/) (2018)
- [Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning](https://mlsanity.com/paper/1612.01887/) (2016)
- [Speed/accuracy trade-offs for modern convolutional object detectors](https://mlsanity.com/paper/1611.10012/) (2016)
- [Neural Baby Talk](https://mlsanity.com/paper/1803.09845/) (2018)
- [Unpaired Image Captioning via Scene Graph Alignments](https://mlsanity.com/paper/1903.10658/) (2019)
- [Unsupervised Image Captioning](https://mlsanity.com/paper/1811.10787/) (2018)
- [Towards Unsupervised Image Captioning with Shared Multimodal Embeddings](https://mlsanity.com/paper/1908.09317/) (2019)
- [Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data](https://mlsanity.com/paper/1803.08314/) (2018)
- [Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data](https://mlsanity.com/paper/1511.05284/) (2015)
- [Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach](https://mlsanity.com/paper/1909.02201/) (2019)
- [Captioning Images with Diverse Objects](https://mlsanity.com/paper/1606.07770/) (2016)
- [Unpaired Image Captioning by Language Pivoting](https://mlsanity.com/paper/1803.05526/) (2018)
- [Compositional Generalization in Image Captioning](https://mlsanity.com/paper/1909.04402/) (2019)
- [Recurrent Relational Memory Network for Unsupervised Image Captioning](https://mlsanity.com/paper/2006.13611/) (2020)

## Built on by (most cited first, 14 of 14)

- [Language Models Can See: Plugging Visual Controls in Text Generation](https://mlsanity.com/paper/2205.02655/) (2022)
- [Multimodal Knowledge Alignment with Reinforcement Learning](https://mlsanity.com/paper/2205.12630/) (2022)
- [MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning](https://mlsanity.com/paper/2308.13218/) (2023)
- [CgT-GAN: CLIP-guided Text GAN for Image Captioning](https://mlsanity.com/paper/2308.12045/) (2023)
- [Object-Centric Unsupervised Image Captioning](https://mlsanity.com/paper/2112.00969/) (2021)
- [Zero-shot Image Captioning by Anchor-augmented Vision-Language Space Alignment](https://mlsanity.com/paper/2211.07275/) (2022)
- [ZeroGen: Zero-shot Multimodal Controllable Text Generation with Multiple Oracles](https://mlsanity.com/paper/2306.16649/) (2023)
- [Detecting Suicide Risk in Online Counseling Services: A Study in a Low-Resource Language](https://mlsanity.com/paper/2209.04830/) (2022)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Emergent Natural Language with Communication Games for Improving Image Captioning Capabilities without Additional Data](https://mlsanity.com/paper/2507.08610/) (2025)
- [Exploring Annotation-free Image Captioning with Retrieval-augmented Pseudo Sentence Generation](https://mlsanity.com/paper/2307.14750/) (2023)
- [Semi-Supervised Image Captioning Considering Wasserstein Graph Matching](https://mlsanity.com/paper/2403.17995/) (2024)
- [AGIC: Attention-Guided Image Captioning to Improve Caption Relevance](https://mlsanity.com/paper/2508.06853/) (2025)
- [TIP and Polish: Text-Image-Prototype Guided Multi-Modal Generation via Commonality-Discrepancy Modeling and Refinement](https://mlsanity.com/paper/2511.21698/) (2025)

## Abstract (arXiv)

Unsupervised image captioning is a challenging task that aims at generating captions without the supervision of image-sentence pairs, but only with images and sentences drawn from different sources and object labels detected from the images. In previous work, pseudo-captions, i.e., sentences that contain the detected object labels, were assigned to a given image. The focus of the previous work was on the alignment of input images and pseudo-captions at the sentence level. However, pseudo-captions contain many words that are irrelevant to a given image. In this work, we investigate the effect of removing mismatched words from image-sentence alignment to determine how they make this task difficult. We propose a simple gating mechanism that is trained to align image features with only the most reliable words in pseudo-captions: the detected object labels. The experimental results show that our proposed method outperforms the previous methods without introducing complex sentence-level learning objectives. Combined with the sentence-level alignment method of previous work, our method further improves its performance. These results confirm the importance of careful alignment in word-level details.