# Image Captioning with Unseen Objects

arXiv: https://arxiv.org/abs/1908.00047 · HTML: https://mlsanity.com/paper/1908.00047/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jul 2019.

## Built on by (most cited first, 6 of 6)

- ["This is my unicorn, Fluffy": Personalizing frozen vision-language representations](https://mlsanity.com/paper/2204.01694/) (2022)
- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [NOC-REK: Novel Object Captioning with Retrieved Vocabulary from External Knowledge](https://mlsanity.com/paper/2203.14499/) (2022)
- [Can Open Domain Question Answering Systems Answer Visual Knowledge Questions?](https://mlsanity.com/paper/2202.04306/) (2022)
- [RCA-NOC: Relative Contrastive Alignment for Novel Object Captioning](https://mlsanity.com/paper/2312.06299/) (2023)
- [Caption Generation on Scenes with Seen and Unseen Object Categories](https://mlsanity.com/paper/2108.06165/) (2021)

## Abstract (arXiv)

Image caption generation is a long standing and challenging problem at the intersection of computer vision and natural language processing. A number of recently proposed approaches utilize a fully supervised object recognition model within the captioning approach. Such models, however, tend to generate sentences which only consist of objects predicted by the recognition models, excluding instances of the classes without labelled training examples. In this paper, we propose a new challenging scenario that targets the image captioning problem in a fully zero-shot learning setting, where the goal is to be able to generate captions of test images containing objects that are not seen during training. The proposed approach jointly uses a novel zero-shot object detection model and a template-based sentence generator. Our experiments show promising results on the COCO dataset.