mlsanityData to 7 Oct 2026

Investigating the Development of Task-Oriented Communication in Vision-Language Models

arXiv 2601.20641 ↗First posted 28 Jan 2026cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

We investigate whether LLM-based agents can develop task-oriented communication protocols that differ from standard natural language in collaborative reasoning tasks. Our focus is on two core properties such task-oriented protocols may exhibit: Efficiency -- conveying task-relevant information more concisely than natural language, and Covertness -- becoming difficult for external observers to interpret, raising concerns about transparency and control. To investigate these aspects, we use a referential-game framework in which vision-language model (VLM) agents communicate, providing a controlled, measurable setting for evaluating language variants. Experiments show that VLMs can develop effective, task-adapted communication patterns. At the same time, they can develop covert protocols that are difficult for humans and external agents to interpret. We also observe spontaneous coordination between similar models without explicitly shared protocols. These findings highlight both the potential and the risks of task-oriented communication, and position referential games as a valuable testbed for future work in this area.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 64 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

30 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023GeoChat: Grounded Large Vision-Language Model for Remote Sensing24 Nov

Other references · 29

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  4. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  5. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  6. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  7. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  8. 2018TarMAC: Targeted Multi-Agent Communicationcited by 201
  9. 2016Multi-Agent Cooperation and the Emergence of (Natural) Languagecited by 181
  10. 2020Emergent Multi-Agent Communication in the Deep Learning Eracited by 168
  11. 2017Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbolscited by 123
  12. 2018Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Inputcited by 102
17 more
  1. 2023Towards image compression with perfect realism at ultra-low bitratescited by 99
  2. 2023Text + Sketch: Image Compression at Ultra Low Ratescited by 63
  3. 2018Emergent Communication through Negotiationcited by 62
  4. 2023Preventing Language Models From Hiding Their Reasoningcited by 62
  5. 2024Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMscited by 52
  6. 2021Emergent Communication of Generalizationscited by 41
  7. 2018Compositional Obverter Communication Learning From Raw Visual Inputcited by 30
  8. 2024MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Modelcited by 29
  9. 2021Interpretable agent communication from scratch (with a generic visual processor emerging on the side)cited by 25
  10. 2024A Review of the Applications of Deep Learning-Based Emergent Communicationcited by 17
  11. 2024Generative Emergent Communication: Large Language Model is a Collective World Modelcited by 15
  12. 2018Learning to Communicate Implicitly By Actionscited by 11
  13. 2020ReferentialGym: A Nomenclature and Framework for Language Emergence & Grounding in (Visual) Referential Gamescited by 10
  14. 2025Shaping Shared Languages: Human and Large Language Models' Inductive Biases in Emergent Communicationcited by 9
  15. 2025Success and Cost Elicit Convention Formation for Efficient Communicationcited by 7
  16. 2024A Network Analysis Approach to Conlang Research Literaturecited by 2
  17. 2026CtD: Composition through Decomposition in Emergent Communicationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026GlossoGen: Emergent Language in Complex Multi-Agent LLM Interactionscited by 1
  2. 2026Despite Instructions: Frontier Agents Improvise Covert Channels at Test Timecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Communication and Verification in LLM Agents towards Collaboration under Information Asymmetry
  2. 2024A Survey on Large Language Model-Based Game Agents
  3. 2023Cooperation on the Fly: Exploring Language Agents for Ad Hoc Teamwork in the Avalon Game
  4. 2024COMMA: A Communicative Multimodal Multi-Agent Benchmark
  5. 2026Collaborative Memory for Multi-Agent VLM Systems
  6. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments
  7. 2026CRAFT: Grounded Multi-Agent Coordination Under Partial Information
  8. 2026Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation