mlsanityData to 7 Oct 2026

Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models

arXiv 2409.09788 ↗First posted 15 Sep 2024cs.CV cs.CLcited by 54

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Despite recent advances demonstrating vision-language models' (VLMs) abilities to describe complex relationships in images using natural language, their capability to quantitatively reason about object sizes and distances remains underexplored. In this work, we introduce a manually annotated benchmark, Q-Spatial Bench, with 271 questions across five categories designed for quantitative spatial reasoning and systematically investigate the performance of state-of-the-art VLMs on this task. Our analysis reveals that reasoning about distances between objects is particularly challenging for SoTA VLMs; however, some VLMs significantly outperform others, with an over 40-point gap between the two best performing models. We also make the surprising observation that the success rate of the top-performing VLM increases by 19 points when a reasoning path using a reference object emerges naturally in the response. Inspired by this observation, we develop a zero-shot prompting technique, SpatialPrompt, that encourages VLMs to answer quantitative spatial questions using reference objects as visual cues. By instructing VLMs to use reference objects in their reasoning paths via SpatialPrompt, Gemini 1.5 Pro, Gemini 1.5 Flash, and GPT-4V improve their success rates by over 40, 20, and 30 points, respectively. We emphasize that these significant improvements are obtained without needing more data, model architectural modifications, or fine-tuning.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 34 references that are in the map. First posted on arXiv 15 Sep 2024.

Builds on

19 of 34 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  2. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct

Other references · 17

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  9. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  10. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  11. 2023Vision-Language Models as Success Detectorscited by 138
  12. 2021Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluationcited by 69
5 more
  1. 2019SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognitioncited by 55
  2. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  3. 2020Rel3D: A Minimally Contrastive Benchmark for Grounding Spatial Relations in 3Dcited by 30
  4. 2021Grounding Predicates through Actionscited by 25
  5. 2020Transferable Task Execution from Pixels through Deep Planning Domain Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 27; 2026: 25.

54 papers.

Sort:

By area:

  1. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  2. 2024LLaMA-Mesh: Unifying 3D Mesh Generation with Language Modelscited by 87
  3. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  4. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 62
  5. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 58
  6. 2025Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Modelscited by 52
  7. 2025LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?cited by 49
  8. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  9. 2025Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learningcited by 38
  10. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  11. 20253D Aware Region Prompted Vision Language Modelcited by 31
  12. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
38 more
  1. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  2. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Modelscited by 24
  3. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 22
  4. 2025SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligencecited by 19
  5. 2026Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?cited by 18
  6. 2025SVQA-R1: Reinforcing Spatial Reasoning in MLLMs via View-Consistent Reward Optimizationcited by 16
  7. 2025LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perceptioncited by 14
  8. 2025Towards Spatial Trace with Reasoning in Vision-Language Models for Roboticscited by 12
  9. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  10. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sightcited by 8
  11. 2025IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigationcited by 5
  12. 2026Allocentric Perceiver: Disentangling Allocentric Reasoning from Egocentric Visual Priors via Frame Instantiationcited by 5
  13. 2026Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoningcited by 5
  14. 2025Aligning Text, Images, and 3D Structure Token-by-Tokencited by 4
  15. 2025Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoningcited by 4
  16. 2024Task-oriented Robotic Manipulation with Vision Language Modelscited by 3
  17. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  18. 2025Scaling Spatial Reasoning in MLLMs through Programmatic Data Synthesiscited by 3
  19. 2025Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Studycited by 2
  20. 2026Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Textcited by 2
  21. 2025Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Taskscited by 1
  22. 2026Physically Guided Visual Mass Estimation from a Single RGB Imagecited by 1
  23. 2026It's Time to Get It Right: Improving Analog Clock Reading and Clock-Hand Spatial Reasoning in Vision-Language Modelscited by 1
  24. 2026HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Modelscited by 1
  25. 2026Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Modelscited by 1
  26. 2026SPACENUM: Revisiting Spatial Numerical Understanding in VLMscited by 1
  27. 2026Grounded 3D-Aware Spatial Vision-Language Modelingcited by 1
  28. 2026Vision-Language Models for Ergonomic Assessment of Manual Lifting Tasks: Estimating Horizontal and Vertical Hand Distances from RGB Videocited by 0
  29. 2026VehAnchor: Metadata-Free Metric Scale Recovery from Vehicle Cues in Aerial Imagerycited by 0
  30. 2026Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structurescited by 0
  31. 2026Spatial Atlas: Compute-Grounded Reasoning for Spatial-Aware Research Agent Benchmarkscited by 0
  32. 2026RoboPIN: Grounded Embodied Reasoning via Pinned Chain-of-Thoughtcited by 0
  33. 2026AeroVerse-SatAgent: UAV-Satellite Collaborative Spatial Reasoning Inspired by the Dual Visual Pathway Theory of Cognitive Neurosciencecited by 0
  34. 2026When Depth Is Better Told Than Shown: Depth-Ordinal Prompting for Vision-Language Spatial Reasoningcited by 0
  35. 2026AlloEgo-VLM: Disambiguating Allocentric and Egocentric Reference Frames in Vision-Language Modelscited by 0
  36. 2026PhysBrain 1.5: From Vision-Language Models to Physical Foundation Modelscited by 0
  37. 2026VABench: Measuring Embodied Spatial Intelligence through Visual Demonstrations, Active Perception, and Metric Controlcited by 0
  38. 2026EmbodiedMind: Adaptive Data Curation and Prefix-Tree Reinforcement Learning for Efficient Embodied Intelligencecited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  2. 2025SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities
  4. 2026SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models
  5. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas
  6. 2026SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?
  7. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
  8. 2025Referring Expressions as a Lens into Spatial Language Grounding in Vision-Language Models