mlsanityData to 7 Oct 2026

Composition, Not Conversation: VLMs Lose the Scene, Not the Thread

arXiv 2609.38368 ↗First posted 29 Sep 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) increasingly reason over visual evidence that is cropped, segmented, retrieved, or revealed over time. Yet most VQA benchmarks present the complete image and question at once. We ask what models lose when the same information is fragmented. We introduce Layered-VQA, with 93 scenes and 300 questions. Each image is decomposed into ordered RGBA layers that exactly recompose the original scene, and each question is annotated with supporting, minimal-sufficient, and distractor layers. We evaluate eleven open-weight VLMs from 3B to 32B parameters and two proprietary models with a scale of 187,200 conversations, graded by 1.74M open-model cross-judgments. We find three consistent failures. Loss in Composition: fragmenting the question has a small effect, but fragmenting the scene substantially reduces accuracy; recomposing the same layers largely restores performance. Oracle Inversion: even oracle-selected sufficient evidence can perform worse than the complete scene. Loss in Grounding: as more evidence is required, grounding degrades much faster than answer accuracy. Together, these results show that having the right visual evidence is not enough. How that evidence is composed and presented determines whether models can use and ground it. The right evidence is not enough: VLMs need the scene it came from.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 51 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

30 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem31 Oct

Other references · 29

  1. 2023Segment Anythingcited by 8,158
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  7. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  8. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  9. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  10. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  11. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  12. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
17 more
  1. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  2. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  3. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  4. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  5. 2016Visual Dialogcited by 377
  6. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  7. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  8. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  9. 2024Transparent Image Layer Diffusion using Latent Transparencycited by 111
  10. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  11. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  12. 2024pix2gestalt: Amodal Segmentation by Synthesizing Wholescited by 70
  13. 2024Needle In A Multimodal Haystackcited by 56
  14. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 53
  15. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 53
  16. 2024MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generationcited by 38
  17. 2024Uncovering the Full Potential of Visual Grounding Methods in VQAcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes
  2. 2026Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act
  3. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  4. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs
  5. 2026VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence
  6. 2026Benchmarking Deflection and Hallucination in Large Vision-Language Models
  7. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
  8. 2024Probing Visual Language Priors in VLMs