mlsanityData to 7 Oct 2026

Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities

arXiv 2410.17385 ↗First posted 22 Oct 2024cs.CL cs.CVcited by 58

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Spatial expressions in situated communication can be ambiguous, as their meanings vary depending on the frames of reference (FoR) adopted by speakers and listeners. While spatial language understanding and reasoning by vision-language models (VLMs) have gained increasing attention, potential ambiguities in these models are still under-explored. To address this issue, we present the COnsistent Multilingual Frame Of Reference Test (COMFORT), an evaluation protocol to systematically assess the spatial reasoning capabilities of VLMs. We evaluate nine state-of-the-art VLMs using COMFORT. Despite showing some alignment with English conventions in resolving ambiguities, our experiments reveal significant shortcomings of VLMs: notably, the models (1) exhibit poor robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to language-specific or culture-specific conventions in cross-lingual tests, as English tends to dominate other languages. With a growing effort to align vision-language models with human cognitive intuitions, we call for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 87 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

48 of 87 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
  2. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  4. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct
  5. 2023Towards Grounded Visual Spatial Reasoning in Multi-Modal Vision Language Models18 Aug
  6. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 42

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  11. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
30 more
  1. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  2. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  3. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  4. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  5. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  6. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  7. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  8. 2022Visual Spatial Reasoningcited by 288
  9. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  10. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  11. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  12. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 118
  13. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  14. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 84
  15. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 75
  16. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  17. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
  18. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  19. 2023Prompting is not a substitute for probability measurements in large language modelscited by 43
  20. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
  21. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  22. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
  23. 2022Transfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoningcited by 32
  24. 2023Towards A Holistic Landscape of Situated Theory of Mind in Large Language Modelscited by 32
  25. 2023Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?cited by 32
  26. 2024Multilingual Diversity Improves Vision-Language Representationscited by 14
  27. 2023World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Modelscited by 10
  28. 2024Ego3DT: Tracking Every 3D Object in Ego-centric Videoscited by 7
  29. 2024Neuro-symbolic Training for Reasoning over Spatial Languagecited by 4
  30. 2024Learning Language Structures through Groundingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 30; 2026: 27.

58 papers.

Sort:

By area:

  1. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  2. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  3. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 62
  4. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 49
  5. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  6. 2025RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patternscited by 24
  7. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Modelscited by 24
  8. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 23
  9. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipescited by 22
  10. 2026Orient Anything V2: Unifying Orientation and Rotation Understandingcited by 22
  11. 2025Is CLIP ideal? No. Can we fix it? Yes!cited by 20
  12. 2025SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Datacited by 19
38 more
  1. 2025The Power of Context: How Multimodality Improves Image Super-Resolutioncited by 15
  2. 2026Everything in Its Place: Benchmarking Spatial Intelligence of Text-to-Image Modelscited by 14
  3. 2025LLM-Based Social Simulations Require a Boundarycited by 11
  4. 2025AutoSpatial: Visual-Language Reasoning for Social Robot Navigation through Efficient Spatial Reasoning Learningcited by 10
  5. 2025EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editingcited by 10
  6. 2025SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagerycited by 10
  7. 2025From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMscited by 10
  8. 2025GenSpace: Benchmarking Spatially-Aware Image Generationcited by 9
  9. 2025From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lenscited by 9
  10. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generationcited by 8
  11. 2025STRIVE: Structured Representation Integrating VLM Reasoning for Efficient Object Navigationcited by 5
  12. 2026Allocentric Perceiver: Disentangling Allocentric Reasoning from Egocentric Visual Priors via Frame Instantiationcited by 5
  13. 2025Vision language models are unreliable at trivial spatial cognitioncited by 4
  14. 2025RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skillscited by 4
  15. 2025LRR-Bench: Left, Right or Rotate? Vision-Language models Still Struggle With Spatial Understanding Taskscited by 4
  16. 2025CubeBench: Diagnosing Interactive, Long-Horizon Spatial Reasoning Under Partial Observationscited by 3
  17. 2026Egocentric Bias in Vision-Language Modelscited by 3
  18. 2025Grounded Misunderstandings in Asymmetric Dialogue: A Perspectivist Annotation Scheme for MapTaskcited by 2
  19. 2025Transductive Visual Programming: Evolving Tool Libraries from Experience for Spatial Reasoningcited by 2
  20. 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Modelscited by 2
  21. 2026Aligning What Vision-Language Models See and Perceive with Adaptive Information Flowcited by 2
  22. 2026ProSR: Process-Shaped Spatial Reasoning for Reliable Chain-of-Thought in VLMscited by 2
  23. 2025GuessBench: Sensemaking Multimodal Creativity in the Wildcited by 1
  24. 2025FoR-SALE: Frame of Reference-guided Spatial Adjustment in LLM-based Diffusion Editingcited by 1
  25. 2025CORA: Consistency-Guided Semi-Supervised Framework for Reasoning Segmentationcited by 1
  26. 2026Cognitively-Inspired Tokens Overcome Egocentric Bias in Multimodal Modelscited by 1
  27. 2026Meanings and Measurements: Multi-Agent Probabilistic Grounding for Vision-Language Navigationcited by 1
  28. 2026When Do Diffusion Models learn to Generate Multiple Objects?cited by 1
  29. 2026Why Far Looks Up: Probing Spatial Representation in Vision-Language Modelscited by 1
  30. 2026MT-EditFlow: Reinforcement Learning for Multi-Turn Image Editing with Flow Matchingcited by 1
  31. 2026OmniView-Space: Reinforcing Spatial Reasoning via Multi-Perspective Spatial Mappingcited by 1
  32. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 0
  33. 2026Vision-Language Models for Ergonomic Assessment of Manual Lifting Tasks: Estimating Horizontal and Vertical Hand Distances from RGB Videocited by 0
  34. 2026OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Modelscited by 0
  35. 2026Visual Prompt Discovery via Semantic Explorationcited by 0
  36. 2026Learning Multi-View Spatial Reasoning from Cross-View Relationscited by 0
  37. 2026Token Warping Helps MLLMs Look from Nearby Viewpointscited by 0
  38. 2026IDEAL-Bench: Indoor Dataset and Evaluation suite for Analyzing 3D Layout reasoningcited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  2. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
  3. 2026SpatialText: A Pure-Text Cognitive Benchmark for Spatial Understanding in Large Language Models
  4. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods
  5. 2025Referring Expressions as a Lens into Spatial Language Grounding in Vision-Language Models
  6. 2025Imagine in Space: Exploring the Frontier of Spatial Intelligence and Reasoning Efficiency in Vision Language Models
  7. 2025Exploring Spatial Language Grounding Through Referring Expressions
  8. 2025Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecture