mlsanityData to 7 Oct 2026

When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputs

arXiv 2605.28346 ↗First posted 27 May 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) are increasingly evaluated for whether they identify the right visual content, but little is known about whether they express such content in a discourse-appropriate form. We address this research gap using information structure (IS), testing whether VLMs distinguish discourse-old Topics from discourse-new Foci in visually grounded question answering. We exploit Hungarian, a language in which Topic and Focus map onto dedicated syntactic positions, making IS choices observable in text. Comparing six VLMs with human participants, we find that models produce IS-relevant constructions, but over-regularise this sensitivity. Under the interacting pressures of discourse status, grammatical role (preference for subject Topics) and definiteness (preference for indefinite Foci), humans choose variable strategies for IS realisation. VLMs, by contrast, collapse onto narrow response templates, resembling mode collapse (Kirk et al., 2024). Our findings suggest that VLM evaluation should look beyond content accuracy to how content is packaged for the discourse.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 49 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

13 of 49 references.

References · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  4. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  5. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  6. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
  7. 2017Using simulation studies to evaluate statistical methodscited by 47
  8. 2025Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challengescited by 31
  9. 2024RuBLiMP: Russian Benchmark of Linguistic Minimal Pairscited by 23
  10. 2022HuSpaCy: an industrial-strength Hungarian natural language processing toolkitcited by 8
  11. 2023Advancing Hungarian Text Processing with HuSpaCy: Efficient and Accurate NLP Pipelinescited by 3
  12. 2020The Role of Verb Semantics in Hungarian Verb-Object Ordercited by 1
1 more
  1. 2025For GPT-4 as with Humans: Information Structure Predicts Acceptability of Long-Distance Dependenciescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  2. 2026Mirage Probes: How Vision Models Fake Visual Understanding
  3. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  4. 2026Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act
  5. 2025Teaching Vision-Language Models to Ask: Resolving Ambiguity in Visual Questions
  6. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?
  7. 2025Questioning the Stability of Visual Question Answering
  8. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs