# TopViewRS: Vision-Language Models as Top-View Spatial Reasoners

arXiv: https://arxiv.org/abs/2406.02537 · HTML: https://mlsanity.com/paper/2406.02537/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 41 of its 62 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 41)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age](https://mlsanity.com/paper/1606.05830/) (2016)
- [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://mlsanity.com/paper/2401.16420/) (2024)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [What does CLIP know about a red circle? Visual prompt engineering for VLMs](https://mlsanity.com/paper/2304.06712/) (2023)
- [Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2311.13165/) (2023)
- [Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step](https://mlsanity.com/paper/2306.14050/) (2023)
- [Graph-Structured Representations for Visual Question Answering](https://mlsanity.com/paper/1609.05600/) (2016)
- [BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs](https://mlsanity.com/paper/2203.04050/) (2022)
- [Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipe](https://mlsanity.com/paper/2209.05324/) (2022)
- [Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships](https://mlsanity.com/paper/2402.12259/) (2024)
- [SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoning](https://mlsanity.com/paper/2104.05832/) (2021)
- [Evaluating Cognitive Maps and Planning in Large Language Models with CogEval](https://mlsanity.com/paper/2309.15129/) (2023)
- [StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts](https://mlsanity.com/paper/2204.08292/) (2022)
- [When Do We Not Need Larger Vision Models?](https://mlsanity.com/paper/2403.13043/) (2024)
- [Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Text](https://mlsanity.com/paper/2307.07696/) (2023)
- [What Makes Multimodal In-Context Learning Work?](https://mlsanity.com/paper/2404.15736/) (2024)
- [Grounding 'Grounding' in NLP](https://mlsanity.com/paper/2106.02192/) (2021)

## Built on by (most cited first, 30 of 49)

- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [Visual Planning: Let's Think Only with Images](https://mlsanity.com/paper/2505.11409/) (2025)
- [Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMs](https://mlsanity.com/paper/2510.24514/) (2025)
- [Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulation](https://mlsanity.com/paper/2508.13998/) (2025)
- [From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation](https://mlsanity.com/paper/2505.08548/) (2025)
- [SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition](https://mlsanity.com/paper/2511.21471/) (2025)
- [Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoning](https://mlsanity.com/paper/2410.16162/) (2024)
- [Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes](https://mlsanity.com/paper/2504.15037/) (2025)
- [V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Models](https://mlsanity.com/paper/2511.16668/) (2025)
- [SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards](https://mlsanity.com/paper/2511.07403/) (2025)
- [TopV-Nav: Unlocking the Top-View Spatial Reasoning Potential of MLLM for Zero-shot Object Navigation](https://mlsanity.com/paper/2411.16425/) (2024)
- [Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs](https://mlsanity.com/paper/2502.16435/) (2025)
- [How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing](https://mlsanity.com/paper/2602.01851/) (2026)
- [Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs](https://mlsanity.com/paper/2506.04220/) (2025)
- [LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight](https://mlsanity.com/paper/2511.20648/) (2025)
- [The Spatial Blindspot of Vision-Language Models](https://mlsanity.com/paper/2601.09954/) (2026)
- [Solar Photovoltaic Assessment with Large Language Model](https://mlsanity.com/paper/2507.19144/) (2025)
- [DARE: Diverse Visual Question Answering with Robustness Evaluation](https://mlsanity.com/paper/2409.18023/) (2024)
- [ROOT: VLM based System for Indoor Scene Understanding and Beyond](https://mlsanity.com/paper/2411.15714/) (2024)
- [OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs](https://mlsanity.com/paper/2606.03890/) (2026)
- [Hypo3D: Exploring Hypothetical Reasoning in 3D](https://mlsanity.com/paper/2502.00954/) (2025)
- [iNews: A Multimodal Dataset for Modeling Personalized Affective Responses to News](https://mlsanity.com/paper/2503.03335/) (2025)
- [Learning 3D Scene Analogies with Neural Contextual Scene Maps](https://mlsanity.com/paper/2503.15897/) (2025)
- [Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoning](https://mlsanity.com/paper/2511.16160/) (2025)
- [Attention in Space: Functional Roles of VLM Heads for Spatial Reasoning](https://mlsanity.com/paper/2603.20662/) (2026)
- [Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Models](https://mlsanity.com/paper/2406.14035/) (2024)
- [TDBench: A Benchmark for Top-Down Image Understanding with Reliability Analysis of Vision-Language Models](https://mlsanity.com/paper/2504.03748/) (2025)
- [DialNav: Multi-turn Dialog Navigation with a Remote Guide](https://mlsanity.com/paper/2509.12894/) (2025)
- [Can Vision-Language Models Solve Visual Math Equations?](https://mlsanity.com/paper/2509.09013/) (2025)

## Abstract (arXiv)

Top-view perspective denotes a typical way in which humans read and reason over different types of maps, and it is vital for localization and navigation of humans as well as of `non-human' agents, such as the ones backed by large Vision-Language Models (VLMs). Nonetheless, spatial reasoning capabilities of modern VLMs remain unattested and underexplored. In this work, we thus study their capability to understand and reason over spatial relations from the top view. The focus on top view also enables controlled evaluations at different granularity of spatial reasoning; we clearly disentangle different abilities (e.g., recognizing particular objects versus understanding their relative positions). We introduce the TopViewRS (Top-View Reasoning in Space) dataset, consisting of 11,384 multiple-choice questions with either realistic or semantic top-view map as visual input. We then use it to study and evaluate VLMs across 4 perception and reasoning tasks with different levels of complexity. Evaluation of 10 representative open- and closed-source VLMs reveals the gap of more than 50% compared to average human performance, and it is even lower than the random baseline in some cases. Although additional experiments show that Chain-of-Thought reasoning can boost model capabilities by 5.82% on average, the overall performance of VLMs remains limited. Our findings underscore the critical need for enhanced model capability in top-view spatial reasoning and set a foundation for further research towards human-level proficiency of VLMs in real-world multimodal tasks.