# Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models

arXiv: https://arxiv.org/abs/2409.09788 · HTML: https://mlsanity.com/paper/2409.09788/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 15 Sep 2024.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 19 of 19)

- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](https://mlsanity.com/paper/2406.01584/) (2024)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning](https://mlsanity.com/paper/2310.12921/) (2023)
- [Vision-Language Models as Success Detectors](https://mlsanity.com/paper/2303.07280/) (2023)
- [Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation](https://mlsanity.com/paper/2102.01818/) (2021)
- [SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognition](https://mlsanity.com/paper/1908.02660/) (2019)
- [SORNet: Spatial Object-Centric Representations for Sequential Manipulation](https://mlsanity.com/paper/2109.03891/) (2021)
- [Rel3D: A Minimally Contrastive Benchmark for Grounding Spatial Relations in 3D](https://mlsanity.com/paper/2012.01634/) (2020)
- [Grounding Predicates through Actions](https://mlsanity.com/paper/2109.14718/) (2021)
- [Transferable Task Execution from Pixels through Deep Planning Domain Learning](https://mlsanity.com/paper/2003.03726/) (2020)

## Built on by (most cited first, 30 of 54)

- [MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence](https://mlsanity.com/paper/2505.23764/) (2025)
- [LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models](https://mlsanity.com/paper/2411.09595/) (2024)
- [MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs](https://mlsanity.com/paper/2503.13111/) (2025)
- [Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views](https://mlsanity.com/paper/2510.18632/) (2025)
- [Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes](https://mlsanity.com/paper/2509.06266/) (2025)
- [Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models](https://mlsanity.com/paper/2505.17015/) (2025)
- [LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?](https://mlsanity.com/paper/2503.19990/) (2025)
- [How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective](https://mlsanity.com/paper/2509.18905/) (2025)
- [Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2510.27606/) (2025)
- [From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation](https://mlsanity.com/paper/2505.08548/) (2025)
- [3D Aware Region Prompted Vision Language Model](https://mlsanity.com/paper/2509.13317/) (2025)
- [Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMs](https://mlsanity.com/paper/2506.21656/) (2025)
- [TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Robotics](https://mlsanity.com/paper/2510.07181/) (2025)
- [SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models](https://mlsanity.com/paper/2509.17664/) (2025)
- [SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence](https://mlsanity.com/paper/2505.17012/) (2025)
- [Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods](https://mlsanity.com/paper/2511.15722/) (2025)
- [Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?](https://mlsanity.com/paper/2602.07055/) (2026)
- [LongPerceptualThoughts: Distilling System-2 Reasoning for System-1 Perception](https://mlsanity.com/paper/2504.15362/) (2025)
- [SVQA-R1: Reinforcing Spatial Reasoning in MLLMs via View-Consistent Reward Optimization](https://mlsanity.com/paper/2506.01371/) (2025)
- [Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics](https://mlsanity.com/paper/2512.13660/) (2025)
- [COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligence](https://mlsanity.com/paper/2512.04563/) (2025)
- [LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight](https://mlsanity.com/paper/2511.20648/) (2025)
- [Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning](https://mlsanity.com/paper/2603.23404/) (2026)
- [IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigation](https://mlsanity.com/paper/2511.17384/) (2025)
- [Allocentric Perceiver: Disentangling Allocentric Reasoning from Egocentric Visual Priors via Frame Instantiation](https://mlsanity.com/paper/2602.05789/) (2026)
- [Task-oriented Robotic Manipulation with Vision Language Models](https://mlsanity.com/paper/2410.15863/) (2024)
- [Aligning Text, Images, and 3D Structure Token-by-Token](https://mlsanity.com/paper/2506.08002/) (2025)
- [Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoning](https://mlsanity.com/paper/2511.16160/) (2025)
- [Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neuroscience](https://mlsanity.com/paper/2505.05515/) (2025)
- [Scaling Spatial Reasoning in MLLMs through Programmatic Data Synthesis](https://mlsanity.com/paper/2512.16237/) (2025)

## Abstract (arXiv)

Despite recent advances demonstrating vision-language models' (VLMs) abilities to describe complex relationships in images using natural language, their capability to quantitatively reason about object sizes and distances remains underexplored. In this work, we introduce a manually annotated benchmark, Q-Spatial Bench, with 271 questions across five categories designed for quantitative spatial reasoning and systematically investigate the performance of state-of-the-art VLMs on this task. Our analysis reveals that reasoning about distances between objects is particularly challenging for SoTA VLMs; however, some VLMs significantly outperform others, with an over 40-point gap between the two best performing models. We also make the surprising observation that the success rate of the top-performing VLM increases by 19 points when a reasoning path using a reference object emerges naturally in the response. Inspired by this observation, we develop a zero-shot prompting technique, SpatialPrompt, that encourages VLMs to answer quantitative spatial questions using reference objects as visual cues. By instructing VLMs to use reference objects in their reasoning paths via SpatialPrompt, Gemini 1.5 Pro, Gemini 1.5 Flash, and GPT-4V improve their success rates by over 40, 20, and 30 points, respectively. We emphasize that these significant improvements are obtained without needing more data, model architectural modifications, or fine-tuning.