mlsanityData to 14 Oct 2026

Spatial Understanding from Videos: Structured Prompts Meet Simulation Data

arXiv 2506.03642 ↗First posted 4 Jun 2025cs.CV cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

Visual-spatial understanding, the ability to infer object relationships and layouts from visual input, is fundamental to downstream tasks such as robotic navigation and embodied interaction. However, existing methods face spatial uncertainty and data scarcity, limiting the 3D spatial reasoning capability of pre-trained vision-language models (VLMs). To address these challenges, we present a unified framework for enhancing 3D spatial reasoning in pre-trained VLMs without modifying their architecture. This framework combines SpatialMind, a structured prompting strategy that decomposes complex scenes and questions into interpretable reasoning steps, with ScanForgeQA, a scalable question-answering dataset built from diverse 3D simulation scenes through an automated construction process designed for fine-tuning. Extensive experiments across multiple benchmarks demonstrate the individual and combined effectiveness of our prompting and fine-tuning strategies, and yield insights that may inspire future research on visual-spatial understanding.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 75 of its 94 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

75 of 94 references.

References · 75

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
38 more
  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  4. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  5. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  6. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  7. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  8. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  10. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  11. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  12. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  13. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  14. 2023Point Transformer V3: Simpler, Faster, Strongercited by 589
  15. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  16. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  17. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  18. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
  19. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 391
  20. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  21. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 380
  22. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  23. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  24. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  25. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  26. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  27. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  28. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 268
  29. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 251
  30. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  31. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 200
  32. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  33. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  34. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 152
  35. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 138
  36. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  37. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 136
  38. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 135

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 23.

28 papers.

Sort:

By area:

  1. 2025Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learningcited by 41
  2. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 33
  3. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 24
  4. 2025SpaceVista: All-Scale Visual Spatial Reasoning from mm to kmcited by 19
  5. 2026ConeSep: Cone-based Robust Noise-Unlearning Compositional Network for Composed Image Retrievalcited by 17
  6. 2026Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Mapscited by 8
  7. 2025Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoningcited by 4
  8. 2026TINA: Text-Free Inversion Attack for Unlearned Text-to-Image Diffusion Modelscited by 4
  9. 2026Attention in Space: Functional Roles of VLM Heads for Spatial Reasoningcited by 4
  10. 2026See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoningcited by 4
  11. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  12. 2025EagleVision: A Dual-Stage Framework with BEV-grounding-based Chain-of-Thought for Spatial Intelligencecited by 3
16 more
  1. 2026LychSim: A Controllable and Interactive Simulation Framework for Vision Researchcited by 3
  2. 2026ViSAGE @ NTIRE 2026 Challenge on Video Saliency Predictioncited by 2
  3. 2026VISTA: Technical Report for the Ego4D Short-Term Object Interaction Anticipation at EgoVis 2026cited by 2
  4. 2026JFAA: Technical Report for the EPIC-KITCHENS-100 Action Anticipation Challenge at EgoVis 2026cited by 2
  5. 2026Reason, Then Re-reason: Cross-view Revisiting Improves Spatial Reasoningcited by 2
  6. 2026SpatialUAV: Benchmarking Spatial Intelligence for Low-Altitude UAV Perception, Collaboration, and Motioncited by 2
  7. 2026OmniView-Space: Reinforcing Spatial Reasoning via Multi-Perspective Spatial Mappingcited by 2
  8. 2026MARS: Technical Report for the CASTLE Challenge at EgoVis 2026cited by 1
  9. 2026Skill-3D: Evolving Scene-Aware Skills for Agentic 3D Spatial Reasoningcited by 1
  10. 2026SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Videocited by 1
  11. 2026GeoAnchor: Collaborative Reasoning via Latent Decomposition for 3D Spatial Understandingcited by 1
  12. 2026Token Warping Helps MLLMs Look from Nearby Viewpointscited by 0
  13. 2026OSGNet with MLLM Reranking @ Ego4D Episodic Memory Challenge 2026cited by 0
  14. 2026SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervisioncited by 0
  15. 2026From Reasoning Failures to Composable Video Spatial Intelligencecited by 0
  16. 2026GeoLatent: Geometry-Guided Latent Structuring with Routed Optimization for 3D Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models
  2. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  3. 2025Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs
  4. 2026HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models
  5. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models
  6. 2026SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?
  7. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D
  8. 2025SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes