# Informative Image Captioning with External Sources of Information

arXiv: https://arxiv.org/abs/1906.08876 · HTML: https://mlsanity.com/paper/1906.08876/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2019.

## Built on by (most cited first, 14 of 14)

- [From General to Specific: Informative Scene Graph Generation via Balance Adjustment](https://mlsanity.com/paper/2108.13129/) (2021)
- [Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph](https://mlsanity.com/paper/2107.11970/) (2021)
- [Underspecification in Scene Description-to-Depiction Tasks](https://mlsanity.com/paper/2210.05815/) (2022)
- [ICECAP: Information Concentrated Entity-aware Image Captioning](https://mlsanity.com/paper/2108.02050/) (2021)
- [CapOnImage: Context-driven Dense-Captioning on Image](https://mlsanity.com/paper/2204.12974/) (2022)
- [Beyond Generic: Enhancing Image Captioning with Real-World Knowledge using Vision-Language Pre-Training Model](https://mlsanity.com/paper/2308.01126/) (2023)
- [Generating image captions with external encyclopedic knowledge](https://mlsanity.com/paper/2210.04806/) (2022)
- [Informative Scene Graph Generation via Debiasing](https://mlsanity.com/paper/2308.05286/) (2023)
- [A Unified Framework for Slot based Response Generation in a Multimodal Dialogue System](https://mlsanity.com/paper/2305.17433/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Towards Multimodal Vision-Language Models Generating Non-Generic Text](https://mlsanity.com/paper/2207.04174/) (2022)
- [Video Summarization: Towards Entity-Aware Captions](https://mlsanity.com/paper/2312.02188/) (2023)
- [Beam-Guided Knowledge Replay for Knowledge-Rich Image Captioning using Vision-Language Model](https://mlsanity.com/paper/2505.23358/) (2025)
- [ReCap: Event-Aware Image Captioning with Article Retrieval and Semantic Gaussian Normalization](https://mlsanity.com/paper/2509.01259/) (2025)

## Abstract (arXiv)

An image caption should fluently present the essential information in a given image, including informative, fine-grained entity mentions and the manner in which these entities interact. However, current captioning models are usually trained to generate captions that only contain common object names, thus falling short on an important "informativeness" dimension. We present a mechanism for integrating image information together with fine-grained labels (assumed to be generated by some upstream models) into a caption that describes the image in a fluent and informative manner. We introduce a multimodal, multi-encoder model based on Transformer that ingests both image features and multiple sources of entity labels. We demonstrate that we can learn to control the appearance of these entity labels in the output, resulting in captions that are both fluent and informative.