# Pragmatically Informative Image Captioning with Character-Level Inference

arXiv: https://arxiv.org/abs/1804.05417 · HTML: https://mlsanity.com/paper/1804.05417/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2018.

## Built on by (most cited first, 30 of 42)

- [FUDGE: Controlled Text Generation With Future Discriminators](https://mlsanity.com/paper/2104.05218/) (2021)
- [From General to Specific: Informative Scene Graph Generation via Balance Adjustment](https://mlsanity.com/paper/2108.13129/) (2021)
- [Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes](https://mlsanity.com/paper/2109.08828/) (2021)
- [From partners to populations: A hierarchical Bayesian account of coordination and convention](https://mlsanity.com/paper/2104.05857/) (2021)
- [Image Retrieval from Contextual Descriptions](https://mlsanity.com/paper/2203.15867/) (2022)
- [Reference-Centric Models for Grounded Collaborative Dialogue](https://mlsanity.com/paper/2109.05042/) (2021)
- [Learning to Mediate Disparities Towards Pragmatic Communication](https://mlsanity.com/paper/2203.13685/) (2022)
- [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://mlsanity.com/paper/2211.08371/) (2022)
- [RSA-Control: A Pragmatics-Grounded Lightweight Controllable Text Generation Framework](https://mlsanity.com/paper/2410.19109/) (2024)
- [Innovations in Neural Data-to-text Generation: A Survey](https://mlsanity.com/paper/2207.12571/) (2022)
- [Dealing with Semantic Underspecification in Multimodal NLP](https://mlsanity.com/paper/2306.05240/) (2023)
- [DisCLIP: Open-Vocabulary Referring Expression Generation](https://mlsanity.com/paper/2305.19108/) (2023)
- [Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind](https://mlsanity.com/paper/2305.19933/) (2023)
- [Critic-Driven Decoding for Mitigating Hallucinations in Data-to-text Generation](https://mlsanity.com/paper/2310.16964/) (2023)
- [CoGen: Learning from Feedback with Coupled Comprehension and Generation](https://mlsanity.com/paper/2408.15992/) (2024)
- [A practical introduction to the Rational Speech Act modeling framework](https://mlsanity.com/paper/2105.09867/) (2021)
- [GLIMPSE: Pragmatically Informative Multi-Document Summarization for Scholarly Reviews](https://mlsanity.com/paper/2406.07359/) (2024)
- [Evaluating Pragmatic Abilities of Image Captioners on A3DS](https://mlsanity.com/paper/2305.12777/) (2023)
- [Generating Pragmatic Examples to Train Neural Program Synthesizers](https://mlsanity.com/paper/2311.05740/) (2023)
- [Switching to Discriminative Image Captioning by Relieving a Bottleneck of Reinforcement Learning](https://mlsanity.com/paper/2212.03230/) (2022)
- [Pragmatic Inference with a CLIP Listener for Contrastive Captioning](https://mlsanity.com/paper/2306.08818/) (2023)
- [LiFi: Lightweight Controlled Text Generation with Fine-Grained Control Codes](https://mlsanity.com/paper/2402.06930/) (2024)
- [Amortizing Pragmatic Program Synthesis with Rankings](https://mlsanity.com/paper/2407.02499/) (2024)
- [Informative Scene Graph Generation via Debiasing](https://mlsanity.com/paper/2308.05286/) (2023)
- [PROGrasp: Pragmatic Human-Robot Communication for Object Grasping](https://mlsanity.com/paper/2309.07759/) (2023)
- [Resilience through Scene Context in Visual Referring Expression Generation](https://mlsanity.com/paper/2404.12289/) (2024)
- [Cognitive Modeling with Scaffolded LLMs: A Case Study of Referential Expression Generation](https://mlsanity.com/paper/2407.03805/) (2024)
- [Communicating with Speakers and Listeners of Different Pragmatic Levels](https://mlsanity.com/paper/2410.05851/) (2024)
- [Collaborative Rational Speech Act: Pragmatic Reasoning for Multi-Turn Dialog](https://mlsanity.com/paper/2507.14063/) (2025)
- [When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluation](https://mlsanity.com/paper/2601.04609/) (2026)

## Abstract (arXiv)

We combine a neural image captioner with a Rational Speech Acts (RSA) model to make a system that is pragmatically informative: its objective is to produce captions that are not merely true but also distinguish their inputs from similar images. Previous attempts to combine RSA with neural image captioning require an inference which normalizes over the entire set of possible utterances. This poses a serious problem of efficiency, previously solved by sampling a small subset of possible utterances. We instead solve this problem by implementing a version of RSA which operates at the level of characters ("a","b","c"...) during the unrolling of the caption. We find that the utterance-level effect of referential captions can be obtained with only character-level decisions. Finally, we introduce an automatic method for testing the performance of pragmatic speaker models, and show that our model outperforms a non-pragmatic baseline as well as a word-level RSA captioner.