# Reasoning About Pragmatics with Neural Listeners and Speakers

arXiv: https://arxiv.org/abs/1604.00562 · HTML: https://mlsanity.com/paper/1604.00562/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2016.

## Built on by (most cited first, 30 of 68)

- [Modeling Strong and Human-Like Gameplay with KL-Regularized Search](https://mlsanity.com/paper/2112.07544/) (2021)
- [Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes](https://mlsanity.com/paper/2109.08828/) (2021)
- [From partners to populations: A hierarchical Bayesian account of coordination and convention](https://mlsanity.com/paper/2104.05857/) (2021)
- [Interactive Learning from Activity Description](https://mlsanity.com/paper/2102.07024/) (2021)
- [Image Retrieval from Contextual Descriptions](https://mlsanity.com/paper/2203.15867/) (2022)
- [How to talk so AI will learn: Instructions, descriptions, and autonomy](https://mlsanity.com/paper/2206.07870/) (2022)
- [Human-like Few-Shot Learning via Bayesian Reasoning over Natural Language](https://mlsanity.com/paper/2306.02797/) (2023)
- [Modeling Cross-Cultural Pragmatic Inference with Codenames Duet](https://mlsanity.com/paper/2306.02475/) (2023)
- [Affection: Learning Affective Explanations for Real-World Visual Data](https://mlsanity.com/paper/2210.01946/) (2022)
- [Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs](https://mlsanity.com/paper/2408.01417/) (2024)
- [Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Personalization](https://mlsanity.com/paper/2306.09299/) (2023)
- [Reference-Centric Models for Grounded Collaborative Dialogue](https://mlsanity.com/paper/2109.05042/) (2021)
- [Computational Language Acquisition with Theory of Mind](https://mlsanity.com/paper/2303.01502/) (2023)
- [Learning to Mediate Disparities Towards Pragmatic Communication](https://mlsanity.com/paper/2203.13685/) (2022)
- [The Emergence of the Shape Bias Results from Communicative Efficiency](https://mlsanity.com/paper/2109.06232/) (2021)
- [Towards More Human-like AI Communication: A Review of Emergent Communication Research](https://mlsanity.com/paper/2308.02541/) (2023)
- [A Framework for Learning to Request Rich and Contextually Useful Information from Humans](https://mlsanity.com/paper/2110.08258/) (2021)
- [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://mlsanity.com/paper/2211.08371/) (2022)
- [Object Recognition as Next Token Prediction](https://mlsanity.com/paper/2312.02142/) (2023)
- [Value of Information: A Framework for Human-Agent Communication](https://mlsanity.com/paper/2601.06407/) (2026)
- [RSA-Control: A Pragmatics-Grounded Lightweight Controllable Text Generation Framework](https://mlsanity.com/paper/2410.19109/) (2024)
- [Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspective](https://mlsanity.com/paper/2305.17760/) (2023)
- [Calibrate your listeners! Robust communication-based training for pragmatic speakers](https://mlsanity.com/paper/2110.05422/) (2021)
- [Dealing with Semantic Underspecification in Multimodal NLP](https://mlsanity.com/paper/2306.05240/) (2023)
- [Emergence of hierarchical reference systems in multi-agent communication](https://mlsanity.com/paper/2203.13176/) (2022)
- [DisCLIP: Open-Vocabulary Referring Expression Generation](https://mlsanity.com/paper/2305.19108/) (2023)
- [Iterative Teacher-Aware Learning](https://mlsanity.com/paper/2110.00137/) (2021)
- [Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Models](https://mlsanity.com/paper/2301.05149/) (2022)
- [Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind](https://mlsanity.com/paper/2305.19933/) (2023)
- [Concept-Best-Matching: Evaluating Compositionality in Emergent Communication](https://mlsanity.com/paper/2403.14705/) (2024)

## Abstract (arXiv)

We present a model for pragmatically describing scenes, in which contrastive behavior results from a combination of inference-driven pragmatics and learned semantics. Like previous learned approaches to language generation, our model uses a simple feature-driven architecture (here a pair of neural "listener" and "speaker" models) to ground language in the world. Like inference-driven approaches to pragmatics, our model actively reasons about listener behavior when selecting utterances. For training, our approach requires only ordinary captions, annotated _without_ demonstration of the pragmatic behavior the model ultimately exhibits. In human evaluations on a referring expression game, our approach succeeds 81% of the time, compared to a 69% success rate using existing techniques.