mlsanityData to 7 Oct 2026

Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects?

arXiv 2605.20448 ↗First posted 19 May 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-language models reliably name objects in a scene, but do they represent the 3D layout those objects inhabit? We introduce a 3,034-sample human-curated benchmark targeting three components of spatial understanding: depth-ordered occlusion (probed via three independent counterfactual operationalisations), optical-geometry inference over visible reflections, and volumetric rearrangement planning. Six frontier and open-weight VLMs, scored by trained annotators on 18,204 responses with no LLM-as-judge, reveal a sharp dissociation: models that plan rearrangements over visible layouts at 53--97% accuracy and rarely violate collision constraints fall to 6--45% on occlusion and below 7% on reflections. An embodied-reasoning model reproduces the same profile. White-box analysis on Qwen3-VL-8B-Thinking localises the failure to the visual-token merger: spatial information recoverable throughout the vision encoder becomes inaccessible after token compression and only stabilises again when clean post-merger activations are patched into the language decoder.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 30 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

18 of 30 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025MindCube: Spatial Mental Modeling from Limited Views26 Jun
  2. 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs5 Feb
  3. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  4. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  6. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 12

  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2018World Modelscited by 1,444
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  6. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  7. 2018Object Hallucination in Image Captioningcited by 661
  8. 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 626
  9. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  10. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 293
  11. 2022Visual Spatial Reasoningcited by 288
  12. 2022ComPhy: Compositional Physical Reasoning of Objects and Events from Videoscited by 61

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes
  2. 2026Why Far Looks Up: Probing Spatial Representation in Vision-Language Models
  3. 2026Separating perception from reasoning in vision-language models: a model-free render ceiling for crystal structures
  4. 2025SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes
  5. 2026When Depth Is Better Told Than Shown: Depth-Ordinal Prompting for Vision-Language Spatial Reasoning
  6. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding
  7. 2026The Spatial Blindspot of Vision-Language Models
  8. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D