# Pragmatic Issue-Sensitive Image Captioning

arXiv: https://arxiv.org/abs/2004.14451 · HTML: https://mlsanity.com/paper/2004.14451/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 15 of 15)

- [Underspecification in Scene Description-to-Depiction Tasks](https://mlsanity.com/paper/2210.05815/) (2022)
- [How to talk so AI will learn: Instructions, descriptions, and autonomy](https://mlsanity.com/paper/2206.07870/) (2022)
- [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://mlsanity.com/paper/2211.08371/) (2022)
- [Dealing with Semantic Underspecification in Multimodal NLP](https://mlsanity.com/paper/2306.05240/) (2023)
- [ANNA: Abstractive Text-to-Image Synthesis with Filtered News Captions](https://mlsanity.com/paper/2301.02160/) (2023)
- [Pragmatic Inference with a CLIP Listener for Contrastive Captioning](https://mlsanity.com/paper/2306.08818/) (2023)
- [Policy Learning with a Language Bottleneck](https://mlsanity.com/paper/2405.04118/) (2024)
- [ContextRef: Evaluating Referenceless Metrics For Image Description Generation](https://mlsanity.com/paper/2309.11710/) (2023)
- [Cognitive Modeling with Scaffolded LLMs: A Case Study of Referential Expression Generation](https://mlsanity.com/paper/2407.03805/) (2024)
- [When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluation](https://mlsanity.com/paper/2601.04609/) (2026)
- [Discourse over Discourse: The Need for an Expanded Pragmatic Focus in Conversational AI](https://mlsanity.com/paper/2304.14543/) (2023)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Color Overmodification Emerges from Data-Driven Learning and Pragmatic Reasoning](https://mlsanity.com/paper/2205.09172/) (2022)
- [Expanding the Set of Pragmatic Considerations in Conversational AI](https://mlsanity.com/paper/2310.18435/) (2023)
- [LLMs Infer Cultural Context but Fail to Apply It When Responding](https://mlsanity.com/paper/2606.17688/) (2026)

## Abstract (arXiv)

Image captioning systems have recently improved dramatically, but they still tend to produce captions that are insensitive to the communicative goals that captions should meet. To address this, we propose Issue-Sensitive Image Captioning (ISIC). In ISIC, a captioning system is given a target image and an issue, which is a set of images partitioned in a way that specifies what information is relevant. The goal of the captioner is to produce a caption that resolves this issue. To model this task, we use an extension of the Rational Speech Acts model of pragmatic language use. Our extension is built on top of state-of-the-art pretrained neural image captioners and explicitly reasons about issues in our sense. We establish experimentally that these models generate captions that are both highly descriptive and issue-sensitive, and we show how ISIC can complement and enrich the related task of Visual Question Answering.