mlsanityData to 14 Oct 2026

Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs

arXiv 2408.01417 ↗First posted 2 Aug 2024cs.CL cs.AI cs.CVcited by 18

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Humans spontaneously use increasingly efficient language as interactions progress, by adapting and forming ad-hoc conventions. This phenomenon has been studied extensively using reference games, showing properties of human language that go beyond relaying intents. It remains unexplored whether multimodal large language models (MLLMs) similarly increase communication efficiency during interactions, and what mechanisms they may adopt for this purpose. We introduce ICCA, an automated framework to evaluate such conversational adaptation as an in-context behavior in MLLMs. We evaluate several state-of-the-art MLLMs, and observe that while they may understand the increasingly efficient language of their interlocutor, they do not spontaneously make their own language more efficient over time. This latter ability can only be elicited in some models (e.g., GPT-4) with heavy-handed prompting. This shows that this property of linguistic interaction does not arise from current training regimes, even though it is a common hallmark of human language. ICCA is available at https://github.com/lil-lab/ICCA.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 46 references that are in the map. First posted on arXiv 2 Aug 2024.

Builds on

22 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 104
  6. 2016Reasoning About Pragmatics with Neural Listeners and Speakerscited by 68
  7. 2017Colors in Context: A Pragmatic Neural Model for Grounded Language Understandingcited by 57
  8. 2017Unified Pragmatic Models for Generating and Following Instructionscited by 55
  9. 2023Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agentscited by 54
  10. 2023Grounding Gaps in Language Model Generationscited by 47
  11. 2019The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialoguecited by 46
  12. 2022Abstract Visual Reasoning with Tangram Shapescited by 43
9 more
  1. 2019Characterizing the dynamics of learning in repeated reference gamescited by 34
  2. 2021Few-shot Language Coordination by Modeling Theory of Mindcited by 32
  3. 2024Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questionscited by 32
  4. 2019Continual adaptation for efficient machine communicationcited by 28
  5. 2020Learning to refer informatively by amortizing pragmatic reasoningcited by 17
  6. 2021Analysis of Language Change in Collaborative Instruction Followingcited by 9
  7. 2022From Rewriting to Remembering: Common Ground for Conversational QA Modelscited by 5
  8. 2024American Sign Language Handshapes Reflect Pressures for Communicative Efficiencycited by 5
  9. 2023Semantic uncertainty guides the extension of conventions to new referentscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 9.

18 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2025QUDsim: Quantifying Discourse Similarities in LLM-Generated Textcited by 13
  3. 2025Measuring How (Not Just Whether) VLMs Build Common Groundcited by 8
  4. 2025LVLMs are Bad at Overhearing Human Referential Communicationcited by 8
  5. 2026LVLMs and Humans Ground Differently in Referential Communicationcited by 8
  6. 2025Post-training for Efficient Communication via Convention Formationcited by 7
  7. 2025Success and Cost Elicit Convention Formation for Efficient Communicationcited by 7
  8. 2025Seeing Beyond the Scene: Enhancing Vision-Language Models with Interactional Reasoningcited by 4
  9. 2026Gesturing Toward Abstraction: Multimodal Convention Formation in Collaborative Physical Taskscited by 4
  10. 2025Characterizing Language Use in a Collaborative Situated Gamecited by 3
  11. 2024EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignmentcited by 2
  12. 2026GlossoGen: Emergent Language in Complex Multi-Agent LLM Interactionscited by 1
6 more
  1. 2026Using Machine Mental Imagery for Representing Common Ground in Situated Dialoguecited by 0
  2. 2026Aligned but Not Partner-Specific: Distinguishing How Multimodal LLM Agents Succeed in Reference Games Without Human-Like Conventionscited by 0
  3. 2026Implicit vs. Explicit Prompting Strategies for LVLMs in Referential Communicationcited by 0
  4. 2026DigitalCoach: Communication and Grounding Gaps in Human and Agentic Computer Use Coachingcited by 0
  5. 2026When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMscited by 0
  6. 2026Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLMs syntactically adapt their language use to their conversational partner
  2. 2025A Survey on Multi-Turn Interaction Capabilities of Large Language Models
  3. 2026Evaluating Large Language Models Abilities for Addressee, Turn-change, and Next Speaker Prediction in Meetings
  4. 2025Beyond Single-Turn: A Survey on Multi-Turn Interactions with Large Language Models
  5. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation
  6. 2026SayNext-Bench: Why Do LLMs Struggle with Next-Utterance Anticipation?
  7. 2025Beyond Words: Multimodal LLM Knows When to Speak
  8. 2026Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap