# Towards Unique and Informative Captioning of Images

arXiv: https://arxiv.org/abs/2009.03949 · HTML: https://mlsanity.com/paper/2009.03949/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Sep 2020.

## Built on by (most cited first, 25 of 25)

- [CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks](https://mlsanity.com/paper/2204.10965/) (2022)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Natural Language Descriptions of Deep Visual Features](https://mlsanity.com/paper/2201.11114/) (2022)
- [Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation](https://mlsanity.com/paper/2303.12112/) (2023)
- [Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning](https://mlsanity.com/paper/2110.01705/) (2021)
- [Revisiting the Role of Language Priors in Vision-Language Models](https://mlsanity.com/paper/2306.01879/) (2023)
- [Human-like Controllable Image Captioning with Verb-specific Semantic Roles](https://mlsanity.com/paper/2103.12204/) (2021)
- [Transparent Human Evaluation for Image Captioning](https://mlsanity.com/paper/2111.08940/) (2021)
- [Identifying Interpretable Subspaces in Image Representations](https://mlsanity.com/paper/2307.10504/) (2023)
- [ReFormer: The Relational Transformer for Image Captioning](https://mlsanity.com/paper/2107.14178/) (2021)
- [Linking Emergent and Natural Languages via Corpus Transfer](https://mlsanity.com/paper/2203.13344/) (2022)
- [Journalistic Guidelines Aware News Image Captioning](https://mlsanity.com/paper/2109.02865/) (2021)
- [Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matching](https://mlsanity.com/paper/2110.02623/) (2021)
- [Rethinking the Reference-based Distinctive Image Captioning](https://mlsanity.com/paper/2207.11118/) (2022)
- [Distinctive Image Captioning via CLIP Guided Group Optimization](https://mlsanity.com/paper/2208.04254/) (2022)
- [No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioning](https://mlsanity.com/paper/2409.03025/) (2024)
- [Switching to Discriminative Image Captioning by Relieving a Bottleneck of Reinforcement Learning](https://mlsanity.com/paper/2212.03230/) (2022)
- [Improving Reference-based Distinctive Image Captioning with Contrastive Rewards](https://mlsanity.com/paper/2306.14259/) (2023)
- [It's Just Another Day: Unique Video Captioning by Discriminative Prompting](https://mlsanity.com/paper/2410.11702/) (2024)
- [Discovering Hidden Visual Concepts Beyond Linguistic Input in Infant Learning](https://mlsanity.com/paper/2501.05205/) (2025)
- [Caption Generation on Scenes with Seen and Unseen Object Categories](https://mlsanity.com/paper/2108.06165/) (2021)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Controllable Image Captioning](https://mlsanity.com/paper/2204.13324/) (2022)
- [Mammo-CLIP Dissect: A Framework for Analysing Mammography Concepts in Vision-Language Models](https://mlsanity.com/paper/2509.21102/) (2025)
- [Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models](https://mlsanity.com/paper/2607.09654/) (2026)

## Abstract (arXiv)

Despite considerable progress, state of the art image captioning models produce generic captions, leaving out important image details. Furthermore, these systems may even misrepresent the image in order to produce a simpler caption consisting of common concepts. In this paper, we first analyze both modern captioning systems and evaluation metrics through empirical experiments to quantify these phenomena. We find that modern captioning systems return higher likelihoods for incorrect distractor sentences compared to ground truth captions, and that evaluation metrics like SPICE can be 'topped' using simple captioning systems relying on object detectors. Inspired by these observations, we design a new metric (SPICE-U) by introducing a notion of uniqueness over the concepts generated in a caption. We show that SPICE-U is better correlated with human judgements compared to SPICE, and effectively captures notions of diversity and descriptiveness. Finally, we also demonstrate a general technique to improve any existing captioning model -- by using mutual information as a re-ranking objective during decoding. Empirically, this results in more unique and informative captions, and improves three different state-of-the-art models on SPICE-U as well as average score over existing metrics.