# Colors in Context: A Pragmatic Neural Model for Grounded Language Understanding

arXiv: https://arxiv.org/abs/1703.10186 · HTML: https://mlsanity.com/paper/1703.10186/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Mar 2017.

## Built on by (most cited first, 30 of 57)

- [ArtEmis: Affective Language for Visual Art](https://mlsanity.com/paper/2101.07396/) (2021)
- [Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons](https://mlsanity.com/paper/2603.02115/) (2026)
- [Inferring Rewards from Language in Context](https://mlsanity.com/paper/2204.02515/) (2022)
- [Grounding 'Grounding' in NLP](https://mlsanity.com/paper/2106.02192/) (2021)
- [Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges](https://mlsanity.com/paper/2502.12378/) (2025)
- [From partners to populations: A hierarchical Bayesian account of coordination and convention](https://mlsanity.com/paper/2104.05857/) (2021)
- [How to talk so AI will learn: Instructions, descriptions, and autonomy](https://mlsanity.com/paper/2206.07870/) (2022)
- [Modeling Cross-Cultural Pragmatic Inference with Codenames Duet](https://mlsanity.com/paper/2306.02475/) (2023)
- [TriCoLo: Trimodal Contrastive Loss for Text to Shape Retrieval](https://mlsanity.com/paper/2201.07366/) (2022)
- [Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs](https://mlsanity.com/paper/2408.01417/) (2024)
- [Reference-Centric Models for Grounded Collaborative Dialogue](https://mlsanity.com/paper/2109.05042/) (2021)
- [Computational Language Acquisition with Theory of Mind](https://mlsanity.com/paper/2303.01502/) (2023)
- [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://mlsanity.com/paper/2211.08371/) (2022)
- [Value of Information: A Framework for Human-Agent Communication](https://mlsanity.com/paper/2601.06407/) (2026)
- [Innovations in Neural Data-to-text Generation: A Survey](https://mlsanity.com/paper/2207.12571/) (2022)
- [Maintaining Common Ground in Dynamic Environments](https://mlsanity.com/paper/2105.14207/) (2021)
- [Analysis of Language Change in Collaborative Instruction Following](https://mlsanity.com/paper/2109.04452/) (2021)
- [Calibrate your listeners! Robust communication-based training for pragmatic speakers](https://mlsanity.com/paper/2110.05422/) (2021)
- [Dealing with Semantic Underspecification in Multimodal NLP](https://mlsanity.com/paper/2306.05240/) (2023)
- [Measuring How (Not Just Whether) VLMs Build Common Ground](https://mlsanity.com/paper/2509.03805/) (2025)
- [Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind](https://mlsanity.com/paper/2305.19933/) (2023)
- [CoGen: Learning from Feedback with Coupled Comprehension and Generation](https://mlsanity.com/paper/2408.15992/) (2024)
- [Are LLMs good pragmatic speakers?](https://mlsanity.com/paper/2411.01562/) (2024)
- [A practical introduction to the Rational Speech Act modeling framework](https://mlsanity.com/paper/2105.09867/) (2021)
- [Shades of confusion: Lexical uncertainty modulates ad hoc coordination in an interactive communication task](https://mlsanity.com/paper/2105.06546/) (2021)
- [Multimodal Color Recommendation in Vector Graphic Documents](https://mlsanity.com/paper/2308.04118/) (2023)
- [Switching to Discriminative Image Captioning by Relieving a Bottleneck of Reinforcement Learning](https://mlsanity.com/paper/2212.03230/) (2022)
- [Pragmatic Inference with a CLIP Listener for Contrastive Captioning](https://mlsanity.com/paper/2306.08818/) (2023)
- [Regularized Conventions: Equilibrium Computation as a Model of Pragmatic Reasoning](https://mlsanity.com/paper/2311.09712/) (2023)
- [Perceptual Structure in the Absence of Grounding for LLMs: The Impact of Abstractedness and Subjectivity in Color Language](https://mlsanity.com/paper/2311.13105/) (2023)

## Abstract (arXiv)

We present a model of pragmatic referring expression interpretation in a grounded communication task (identifying colors from descriptions) that draws upon predictions from two recurrent neural network classifiers, a speaker and a listener, unified by a recursive pragmatic reasoning framework. Experiments show that this combined pragmatic model interprets color descriptions more accurately than the classifiers from which it is built, and that much of this improvement results from combining the speaker and listener perspectives. We observe that pragmatic reasoning helps primarily in the hardest cases: when the model must distinguish very similar colors, or when few utterances adequately express the target color. Our findings make use of a newly-collected corpus of human utterances in color reference games, which exhibit a variety of pragmatic behaviors. We also show that the embedded speaker model reproduces many of these pragmatic behaviors.