mlsanityData to 14 Oct 2026

Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Models

arXiv 2503.17349 ↗First posted 21 Mar 2025cs.CVcited by 37

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Vision Language Models (VLMs) excel at identifying and describing objects but often fail at spatial reasoning. We study why VLMs, such as LLaVA, underutilize spatial cues despite having positional encodings and spatially rich vision encoder features. Our analysis reveals a key imbalance: vision token embeddings have much larger norms than text tokens, suppressing LLM's position embedding. To expose this mechanism, we developed three interpretability tools: (1) the Position Sensitivity Index, which quantifies reliance on token order, (2) the Cross Modality Balance, which reveals attention head allocation patterns, and (3) a RoPE Sensitivity probe, which measures dependence on rotary positional embeddings. These tools uncover that vision tokens and system prompts dominate attention. We validated our mechanistic understanding through targeted interventions that predictably restore positional sensitivity. These findings reveal previously unknown failure modes in multimodal attention and demonstrate how interpretability analysis can guide principled improvements.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 35 of its 43 references that are in the map. First posted on arXiv 21 Mar 2025.

Builds on

35 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas3 Mar
  2. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  4. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct

Other references · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2015VQA: Visual Question Answeringcited by 2,506
19 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  4. 2019Root Mean Square Layer Normalizationcited by 1,480
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  6. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  8. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  9. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  10. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  11. 2023Linear Representations of Sentiment in Large Language Modelscited by 163
  12. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  13. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  14. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 122
  15. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 98
  16. 2024Attention Heads of Large Language Models: A Surveycited by 67
  17. 2024Do Vision and Language Encoders Represent the World Similarly?cited by 44
  18. 2023What Makes for Good Visual Tokenizers for Large Language Models?cited by 42
  19. 2022MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answeringcited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 23.

37 papers.

Sort:

By area:

  1. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  2. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 41
  3. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  4. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 33
  5. 2026Thinking with Blueprints: Assisting Vision-Language Models in Spatial Reasoning via Structured Object Representationcited by 21
  6. 2026Large Language Model Reasoning Failurescited by 21
  7. 2026SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoningcited by 15
  8. 2025CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMscited by 14
  9. 2025Rethinking Causal Mask Attention for Vision-Language Inferencecited by 6
  10. 2025The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Losscited by 5
  11. 2026Attention in Space: Functional Roles of VLM Heads for Spatial Reasoningcited by 4
  12. 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoningcited by 4
25 more
  1. 2025CubeBench: Diagnosing Interactive, Long-Horizon Spatial Reasoning Under Partial Observationscited by 3
  2. 2026VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observationscited by 3
  3. 2025Beyond the Vision Encoder: Identifying and Mitigating Spatial Bias in Large Vision-Language Modelscited by 2
  4. 2025Mitigating Coordinate Prediction Bias from Positional Encoding Failurescited by 2
  5. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  6. 20263D Primitives are a Spatial Language for VLMscited by 2
  7. 2026See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understandingcited by 2
  8. 2026AlloSpatial: Agentic Harness Framework for Spatial Reasoning in Foundation Modelscited by 2
  9. 2026Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoningcited by 2
  10. 2026Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindnesscited by 1
  11. 2026World2Mind: Cognition Toolkit for Allocentric Spatial Reasoning in Foundation Modelscited by 1
  12. 2026Why Do Vision Language Models Struggle To Recognize Human Emotions?cited by 1
  13. 2026HyCal: A Training-Free Prototype Calibration Method for Cross-Discipline Few-Shot Class-Incremental Learningcited by 1
  14. 2026Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Modelscited by 1
  15. 2025Where, Not What: Compelling Video LLMs to Learn Geometric Causality for 3D-Groundingcited by 0
  16. 2025A Multi-Modal Neuro-Symbolic Approach for Spatial Reasoning-Based Visual Grounding in Roboticscited by 0
  17. 2025An Evaluation of Interleaved Instruction Tuning on Semantic Reasoning Performance in an Audio MLLMcited by 0
  18. 2026The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Modelscited by 0
  19. 2026GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Groundingcited by 0
  20. 2026TestHallVQA: Exploring LVLMs' Document-Level Reasoning under Redundant Contexts from Scientific Examscited by 0
  21. 2026Geometric Encoding for Spatial Reasoning in Vision-Language Modelscited by 0
  22. 2026Who Is Left of Whom? Tracing Spatial Evidence and Role Binding in Relative-Position Reasoningcited by 0
  23. 2026SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Modelscited by 0
  24. 2026RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failurescited by 0
  25. 2026How Vision Transformers Represent Position: A Mechanistic Comparison of Absolute and Rotary Positional Encodingscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA
  2. 2026Mechanisms of Object Localization in Vision-Language Models
  3. 2026The Spatial Blindspot of Vision-Language Models
  4. 2025Investigating Spatial Attention Bias in Vision-Language Models
  5. 2026Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models
  6. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
  7. 2026Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptions
  8. 2026Why Far Looks Up: Probing Spatial Representation in Vision-Language Models