mlsanityData to 14 Oct 2026

Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes

arXiv 2509.06266 ↗First posted 8 Sep 2025cs.CVcited by 63

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Understanding 3D spatial relationships remains a major limitation of current Vision-Language Models (VLMs). Prior work has addressed this issue by creating spatial question-answering (QA) datasets based on single images or indoor videos. However, real-world embodied AI agents such as robots and self-driving cars typically rely on ego-centric, multi-view observations. To this end, we introduce Ego3D-Bench, a new benchmark designed to evaluate the spatial reasoning abilities of VLMs using ego-centric, multi-view outdoor data. Ego3D-Bench comprises over 8,600 QA pairs, created with significant involvement from human annotators to ensure quality and diversity. We benchmark 16 SOTA VLMs, including GPT-4o, Gemini1.5-Pro, InternVL3, and Qwen2.5-VL. Our results reveal a notable performance gap between human level scores and VLM performance, highlighting that current VLMs still fall short of human level spatial understanding. To bridge this gap, we propose Ego3D-VLM, a post-training framework that enhances 3D spatial reasoning of VLMs. Ego3D-VLM generates cognitive map based on estimated global 3D coordinates, resulting in 12% average improvement on multi-choice QA and 56% average improvement on absolute distance estimation. Ego3D-VLM is modular and can be integrated with any existing VLM. Together, Ego3D-Bench and Ego3D-VLM offer valuable tools for advancing toward human level spatial understanding in real-world, multi-view environments.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 52 references that are in the map. First posted on arXiv 8 Sep 2025.

Builds on

40 of 52 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence29 May
  2. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models27 May
  3. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  4. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models15 Sep
  5. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models21 Jun
  6. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  7. 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models4 Apr
  8. 2024SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors18 Mar
  9. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  10. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct
  11. 20233D Concept Learning and Reasoning from Multi-View Images20 Mar
  12. 2022Visual Spatial Reasoning30 Apr

Other references · 28

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  4. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  6. 2019Argoverse: 3D Tracking and Forecasting with Rich Mapscited by 732
  7. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  8. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  9. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  10. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  11. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 183
  12. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 181
16 more
  1. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 169
  2. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 147
  3. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 136
  4. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 114
  5. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  6. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Modelscited by 107
  7. 2024ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehiclescited by 93
  8. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  9. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 59
  10. 2021SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistencycited by 48
  11. 2025UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spacescited by 36
  12. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 27
  13. 2024RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentationcited by 11
  14. 2024GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generationcited by 6
  15. 2025CASP: Compression of Large Multimodal Models Based on Attention Sparsitycited by 5
  16. 2023PoseGen: Learning to Generate 3D Human Pose Dataset with NeRFcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 48.

63 papers.

Sort:

By area:

  1. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 65
  2. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 23
  3. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenescited by 20
  4. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 20
  5. 2026RHO: Your Coding Agent is Secretly a Roboticistcited by 20
  6. 2026Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?cited by 19
  7. 2025From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMscited by 12
  8. 2025SpatialTree: How Spatial Abilities Branch Out in MLLMscited by 12
  9. 2026Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D Worldcited by 12
  10. 2025SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagerycited by 11
  11. 2026CityCube: Benchmarking Cross-view Spatial Reasoning on Vision-Language Models in Urban Environmentscited by 11
  12. 2026AgentVLN: Towards Agentic Vision-and-Language Navigationcited by 9
38 more
  1. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Drivingcited by 9
  2. 2026Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptionscited by 7
  3. 2026SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Taskscited by 7
  4. 2025Toward Ambulatory Vision: Learning Visually-Grounded Active View Selectioncited by 6
  5. 2026Vesta: A Generalist Embodied Reasoning Modelcited by 5
  6. 2025Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoningcited by 4
  7. 2025EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoningcited by 4
  8. 2026The Dual Mechanisms of Spatial Variable Binding in Vision-Language Modelscited by 4
  9. 2026See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoningcited by 4
  10. 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoningcited by 4
  11. 2025MonoSR: Open-Vocabulary Spatial Reasoning from Monocular Imagescited by 3
  12. 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modelingcited by 3
  13. 2025Reasoning Path and Latent State Analysis for Multi-view Visual Spatial Reasoning: A Cognitive Science Perspectivecited by 3
  14. 2025OpenView: Empowering MLLMs with Out-of-view VQAcited by 3
  15. 2026From Correspondence to Actions: Human-Like Multi-Image Spatial Reasoning in Multi-modal Large Language Modelscited by 3
  16. 2026VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observationscited by 3
  17. 2026Are VLMs Lost Between Sky and Space? LinkS2Bench for UAV-Satellite Dynamic Cross-View Spatial Intelligencecited by 3
  18. 2026OpenWorldLib: A Unified Codebase and Definition of Advanced World Modelscited by 3
  19. 2025Enhancing Vision-Language Models for Autonomous Driving through Task-Specific Prompting and Spatial Reasoningcited by 2
  20. 2025From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Modelscited by 2
  21. 2026WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigationcited by 2
  22. 2026Communicating about Space: Language-Mediated Spatial Integration Across Partial Viewscited by 2
  23. 2026EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Drivingcited by 2
  24. 2026DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Drivingcited by 2
  25. 2026SATURN: Symbolic Spatial Reasoning for Multi-Perspective Groundingcited by 2
  26. 2026SSMNBench: Diagnosing Image-based Cross-View Human-Object Understanding via Single-View Sufficiency and Multi-View Necessitycited by 2
  27. 2026CPPO: Contrastive Perception Policy Optimization for VLM Agentscited by 1
  28. 2026Predicting Camera Pose from Perspective Descriptions for Spatial Reasoningcited by 1
  29. 2026ViewFusion: Structured Spatial Thinking Chains for Multi-View Reasoningcited by 1
  30. 2026It's Time to Get It Right: Improving Analog Clock Reading and Clock-Hand Spatial Reasoning in Vision-Language Modelscited by 1
  31. 2026Perceive-then-Plan: Layout-as-Policy for Monocular 3D Scene Layout Estimationcited by 1
  32. 2026GeoAnchor: Collaborative Reasoning via Latent Decomposition for 3D Spatial Understandingcited by 1
  33. 2026SpatialQuery: Benchmarking Geometry-Grounded Multi-Instance Spatial Reasoning in Vision-Language Modelscited by 1
  34. 2026m2sv: A Scalable Benchmark for Map-to-Street-View Spatial Reasoningcited by 0
  35. 2026BEVLM: Distilling Semantic Knowledge from LLMs into Bird's-Eye View Representationscited by 0
  36. 2026BEACON: Language-Conditioned Navigation Affordance Prediction under Occlusioncited by 0
  37. 2026PhysQuantAgent: An Inference Pipeline of Mass Estimation for Vision-Language Modelscited by 0
  38. 2026Feeling the Space: Egomotion-Aware Video Representation for Efficient and Accurate 3D Scene Understandingcited by 0

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data
  2. 2025Out of Sight, Not Out of Context? Egocentric Spatial Reasoning in VLMs Across Disjoint Frames
  3. 2025NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Driving
  4. 2025Open3D-VQA: A Benchmark for Comprehensive Spatial Reasoning with Multimodal Large Language Model in Open Space
  5. 2026MindEdit-Bench: Benchmarking Object-Level Counterfactual Spatial Reasoning in VLMs from In-the-Wild Photos
  6. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Data
  7. 2025InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models
  8. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D