mlsanityData to 7 Oct 2026

Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas

arXiv 2503.01773 ↗First posted 3 Mar 2025cs.CLcited by 93

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Large Vision Language Models (VLMs) have long struggled with spatial reasoning tasks. Surprisingly, even simple spatial reasoning tasks, such as recognizing "under" or "behind" relationships between only two objects, pose significant challenges for current VLMs. In this work, we study the spatial reasoning challenge from the lens of mechanistic interpretability, diving into the model's internal states to examine the interactions between image and text tokens. By tracing attention distribution over the image through out intermediate layers, we observe that successful spatial reasoning correlates strongly with the model's ability to align its attention distribution with actual object locations, particularly differing between familiar and unfamiliar spatial relationships. Motivated by these findings, we propose ADAPTVIS based on inference-time confidence scores to sharpen the attention on highly relevant regions when confident, while smoothing and broadening the attention window to consider a wider context when confidence is lower. This training-free decoding method shows significant improvement (e.g., up to a 50 absolute point improvement) on spatial reasoning benchmarks such as WhatsUp and VSR with negligible cost. We make code and data publicly available for research purposes at https://github.com/shiqichen17/AdaptVis.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 29 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

23 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  2. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct
  3. 2022Visual Spatial Reasoning30 Apr

Other references · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
8 more
  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  2. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  3. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  4. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  5. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  6. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82
  7. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 70
  8. 2024In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigationcited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 31; 2026: 62.

93 papers.

Sort:

By area:

  1. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  2. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 48
  3. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  4. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 40
  5. 2025Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Modelscited by 35
  6. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  7. 2025Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecturecited by 30
  8. 2025DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learningcited by 23
  9. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 22
  10. 2026Thinking with Blueprints: Assisting Vision-Language Models in Spatial Reasoning via Structured Object Representationcited by 21
  11. 2026TwinBrainVLA: Unleashing the Potential of Generalist VLMs for Embodied Tasks via Asymmetric Mixture-of-Transformerscited by 14
  12. 2025CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMscited by 13
38 more
  1. 2026Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Modelscited by 11
  2. 2026PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Modelscited by 10
  3. 2025Mind Meets Space: Rethinking Agentic Spatial Intelligence from a Neuroscience-inspired Perspectivecited by 7
  4. 2025V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Modelscited by 7
  5. 2026CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoningcited by 7
  6. 2025Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attentioncited by 6
  7. 2025[De|Re]constructing VLMs' Reasoning in Countingcited by 5
  8. 2025InfiniBench: Infinite Benchmarking for Visual Spatial Reasoning with Customizable Scene Complexitycited by 5
  9. 2025Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Explorationcited by 5
  10. 2026Video-MSR: Benchmarking Multi-hop Spatial Reasoning Capabilities of MLLMscited by 5
  11. 2026Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddingscited by 5
  12. 2025Caption This, Reason That: VLMs Caught in the Middlecited by 4
  13. 2026Pathways of Visual Information Flow in Vision-Language Modelscited by 4
  14. 2025GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generationcited by 3
  15. 2025CubeBench: Diagnosing Interactive, Long-Horizon Spatial Reasoning Under Partial Observationscited by 3
  16. 2025Spatial4D-Bench: A Versatile 4D Spatial Intelligence Benchmarkcited by 3
  17. 2026SCALE: Self-uncertainty Conditioned Adaptive Looking and Execution for Vision-Language-Action Modelscited by 3
  18. 2026VERA: Identifying and Leveraging Visual Evidence Retrieval Heads in Long-Context Understandingcited by 3
  19. 2026Attention in Space: Functional Roles of VLM Heads for Spatial Reasoningcited by 3
  20. 2026The Dual Mechanisms of Spatial Variable Binding in Vision-Language Modelscited by 3
  21. 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoningcited by 3
  22. 2026Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modelingcited by 3
  23. 2025Cross-Layer Vision Smoothing: Enhancing Visual Understanding via Sustained Focus on Key Objects in Large Vision-Language Modelscited by 2
  24. 2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2
  25. 2025Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perceptioncited by 2
  26. 2026EAGLE: Expert-Augmented Attention Guidance for Tuning-Free Industrial Anomaly Detection in Multimodal Large Language Modelscited by 2
  27. 2026Probing Visual Concepts in Lightweight Vision-Language Models for Automated Drivingcited by 2
  28. 2026WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigationcited by 2
  29. 2026VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observationscited by 2
  30. 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Modelscited by 2
  31. 2026How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoningcited by 2
  32. 2026Aligning What Vision-Language Models See and Perceive with Adaptive Information Flowcited by 2
  33. 20263D Primitives are a Spatial Language for VLMscited by 2
  34. 2026Gaze Heads: How VLMs Look at What They Describecited by 2
  35. 2026A Closer Look at Failure Modes in Temporal Understanding of Large Audio-Language Modelscited by 2
  36. 2026Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Frameworkcited by 2
  37. 2025General-Purpose Robotic Navigation via LVLM-Orchestrated Perception, Reasoning, and Actingcited by 1
  38. 2025Imagine in Space: Exploring the Frontier of Spatial Intelligence and Reasoning Efficiency in Vision Language Modelscited by 1

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  2. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
  3. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models
  4. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes
  5. 2025SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data
  6. 2026ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Models
  7. 2025Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs
  8. 2026Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA