# Object Counts! Bringing Explicit Detections Back into Image Captioning

arXiv: https://arxiv.org/abs/1805.00314 · HTML: https://mlsanity.com/paper/1805.00314/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2018.

## Built on by (most cited first, 5 of 5)

- [MSCTD: A Multimodal Sentiment Chat Translation Dataset](https://mlsanity.com/paper/2202.13645/) (2022)
- [Belief Revision based Caption Re-ranker with Visual Semantic Information](https://mlsanity.com/paper/2209.08163/) (2022)
- [Visual Semantic Relatedness Dataset for Image Captioning](https://mlsanity.com/paper/2301.08784/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Word to Sentence Visual Semantic Similarity for Caption Generation: Lessons Learned](https://mlsanity.com/paper/2209.12817/) (2022)

## Abstract (arXiv)

The use of explicit object detectors as an intermediate step to image captioning - which used to constitute an essential stage in early work - is often bypassed in the currently dominant end-to-end approaches, where the language model is conditioned directly on a mid-level image embedding. We argue that explicit detections provide rich semantic information, and can thus be used as an interpretable representation to better understand why end-to-end image captioning systems work well. We provide an in-depth analysis of end-to-end image captioning by exploring a variety of cues that can be derived from such object detections. Our study reveals that end-to-end image captioning systems rely on matching image representations to generate captions, and that encoding the frequency, size and position of objects are complementary and all play a role in forming a good image representation. It also reveals that different object categories contribute in different ways towards image captioning.