mlsanityData to 14 Oct 2026

Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mind

arXiv 2305.19933 ↗First posted 31 May 2023cs.CL cs.AI cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Dialogue participants may have varying levels of knowledge about the topic under discussion. In such cases, it is essential for speakers to adapt their utterances by taking their audience into account. Yet, it is an open question how such adaptation can be modelled in computational agents. In this paper, we model a visually grounded referential game between a knowledgeable speaker and a listener with more limited visual and linguistic experience. Inspired by psycholinguistic theories, we endow our speaker with the ability to adapt its referring expressions via a simulation module that monitors the effectiveness of planned utterances from the listener's perspective. We propose an adaptation mechanism building on plug-and-play approaches to controlled language generation, where utterance generation is steered on the fly by the simulator without finetuning the speaker's underlying language model. Our results and analyses show that our approach is effective: the speaker's utterances become closer to the listener's domain of expertise, which leads to higher communicative success.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 35 of its 66 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

35 of 66 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  10. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  11. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  12. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
23 more
  1. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  2. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  3. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  4. 2018Machine Theory of Mindcited by 226
  5. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  6. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 161
  7. 2016A Joint Speaker-Listener-Reinforcer Model for Referring Expressionscited by 74
  8. 2016Reasoning About Pragmatics with Neural Listeners and Speakerscited by 68
  9. 2020Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learningcited by 66
  10. 2021PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domainscited by 61
  11. 2017Colors in Context: A Pragmatic Neural Model for Grounded Language Understandingcited by 57
  12. 2021A Plug-and-Play Method for Controlled Text Generationcited by 56
  13. 2019The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialoguecited by 46
  14. 2017Context-aware Captions from Context-agnostic Supervisioncited by 43
  15. 2018Pragmatically Informative Image Captioning with Character-Level Inferencecited by 42
  16. 2021Few-shot Language Coordination by Modeling Theory of Mindcited by 32
  17. 2019Continual adaptation for efficient machine communicationcited by 28
  18. 2020Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contextscited by 22
  19. 2021Reference-Centric Models for Grounded Collaborative Dialoguecited by 15
  20. 2022Learning to Mediate Disparities Towards Pragmatic Communicationcited by 14
  21. 2019Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categoriescited by 8
  22. 2019Modeling Conceptual Understanding in Image Reference Gamescited by 4
  23. 2022Know your audience: specializing grounded language models with listener subtractioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2025Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challengescited by 31
  2. 2025Large Language Models as Theory of Mind Aware Generative Agents with Counterfactual Reflectioncited by 13
  3. 2024Instruct Once, Chat Consistently in Multiple Rounds: An Efficient Tuning Framework for Dialoguecited by 11
  4. 2024Explicit Modelling of Theory of Mind for Belief Prediction in Nonverbal Social Interactionscited by 6
  5. 2025Direct Preference Optimization for Adaptive Concept-based Explanationscited by 0
  6. 2026Personalizing MLLMs via Reinforced Multimodal Reference Gamecited by 0
  7. 2026Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretationcited by 0