# Object-Centric Unsupervised Image Captioning arXiv: https://arxiv.org/abs/2112.00969 ยท HTML: https://mlsanity.com/paper/2112.00969/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 36 of its 45 references that are in the map. - First posted on arXiv 2 Dec 2021. ## Builds on (most cited first, 30 of 36) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014) - [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016) - [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014) - [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015) - [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015) - [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017) - [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014) - [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016) - [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019) - [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017) - [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019) - [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016) - [Connecting Vision and Language with Localized Narratives](https://mlsanity.com/paper/1912.03098/) (2019) - [DenseCap: Fully Convolutional Localization Networks for Dense Captioning](https://mlsanity.com/paper/1511.07571/) (2015) - [Unpaired Image Captioning via Scene Graph Alignments](https://mlsanity.com/paper/1903.10658/) (2019) - [COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrieval](https://mlsanity.com/paper/1805.08661/) (2018) - [Unsupervised Image Captioning](https://mlsanity.com/paper/1811.10787/) (2018) - [Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions](https://mlsanity.com/paper/1811.10652/) (2018) - [Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speech](https://mlsanity.com/paper/1805.12589/) (2018) - [Towards Unsupervised Image Captioning with Shared Multimodal Embeddings](https://mlsanity.com/paper/1908.09317/) (2019) - [Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data](https://mlsanity.com/paper/1511.05284/) (2015) - [Fluency-Guided Cross-Lingual Image Captioning](https://mlsanity.com/paper/1708.04390/) (2017) - [Captioning Images with Diverse Objects](https://mlsanity.com/paper/1606.07770/) (2016) - [Unpaired Image Captioning by Language Pivoting](https://mlsanity.com/paper/1803.05526/) (2018) - [Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning](https://mlsanity.com/paper/1510.03519/) (2015) - [Recurrent Relational Memory Network for Unsupervised Image Captioning](https://mlsanity.com/paper/2006.13611/) (2020) ## Built on by (most cited first, 9 of 9) - [Exploring Discrete Diffusion Models for Image Captioning](https://mlsanity.com/paper/2211.11694/) (2022) - [CgT-GAN: CLIP-guided Text GAN for Image Captioning](https://mlsanity.com/paper/2308.12045/) (2023) - [MedRAT: Unpaired Medical Report Generation via Auxiliary Tasks](https://mlsanity.com/paper/2407.03919/) (2024) - [Semi-Supervised Image Captioning by Adversarially Propagating Labeled Data](https://mlsanity.com/paper/2301.11174/) (2023) - [FIRST: A Million-Entry Dataset for Text-Driven Fashion Synthesis and Design](https://mlsanity.com/paper/2311.07414/) (2023) - [MedCycle: Unpaired Medical Report Generation via Cycle-Consistency](https://mlsanity.com/paper/2403.13444/) (2024) - [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024) - [FaceGemma: Enhancing Image Captioning with Facial Attributes for Portrait Images](https://mlsanity.com/paper/2309.13601/) (2023) - [TIP and Polish: Text-Image-Prototype Guided Multi-Modal Generation via Commonality-Discrepancy Modeling and Refinement](https://mlsanity.com/paper/2511.21698/) (2025) ## Abstract (arXiv) Image captioning is a longstanding problem in the field of computer vision and natural language processing. To date, researchers have produced impressive state-of-the-art performance in the age of deep learning. Most of these state-of-the-art, however, requires large volume of annotated image-caption pairs in order to train their models. When given an image dataset of interests, practitioner needs to annotate the caption for each image in the training set and this process needs to happen for each newly collected image dataset. In this paper, we explore the task of unsupervised image captioning which utilizes unpaired images and texts to train the model so that the texts can come from different sources than the images. A main school of research on this topic that has been shown to be effective is to construct pairs from the images and texts in the training set according to their overlap of objects. Unlike in the supervised setting, these constructed pairings are however not guaranteed to have fully overlapping set of objects. Our work in this paper overcomes this by harvesting objects corresponding to a given sentence from the training set, even if they don't belong to the same image. When used as input to a transformer, such mixture of objects enables larger if not full object coverage, and when supervised by the corresponding sentence, produced results that outperform current state of the art unsupervised methods by a significant margin. Building upon this finding, we further show that (1) additional information on relationship between objects and attributes of objects also helps in boosting performance; and (2) our method also extends well to non-English image captioning, which usually suffers from a scarcer level of annotations. Our findings are supported by strong empirical results. Our code is available at https://github.com/zihangm/obj-centric-unsup-caption.