mlsanityData to 14 Oct 2026

SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities

arXiv 2401.12168 ↗First posted 22 Jan 2024cs.CV cs.CL cs.LGcited by 706

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Understanding and reasoning about spatial relationships is a fundamental capability for Visual Question Answering (VQA) and robotics. While Vision Language Models (VLM) have demonstrated remarkable performance in certain VQA benchmarks, they still lack capabilities in 3D spatial reasoning, such as recognizing quantitative relationships of physical objects like distances or size differences. We hypothesize that VLMs' limited spatial reasoning capability is due to the lack of 3D spatial knowledge in training data and aim to solve this problem by training VLMs with Internet-scale spatial reasoning data. To this end, we present a system to facilitate this approach. We first develop an automatic 3D spatial VQA data generation framework that scales up to 2 billion VQA examples on 10 million real-world images. We then investigate various factors in the training recipe, including data quality, training pipeline, and VLM architecture. Our work features the first internet-scale 3D spatial reasoning dataset in metric space. By training a VLM on such data, we significantly enhance its ability on both qualitative and quantitative spatial VQA. Finally, we demonstrate that this VLM unlocks novel downstream applications in chain-of-thought spatial reasoning and robotics due to its quantitative estimation capability. Project website: https://spatial-vlm.github.io/

Key facts. Cited by 706 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 56 of its 72 references that are in the map. First posted on arXiv 22 Jan 2024.

Builds on

56 of 72 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023RoboVQA: Multimodal Long-Horizon Reasoning for Robotics1 Nov
  2. 2022Visual Spatial Reasoning30 Apr

Other references · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Segment Anythingcited by 8,234
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2017Mask R-CNNcited by 7,277
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
38 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  4. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  5. 2023PaLM 2 Technical Reportcited by 1,162
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  8. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  9. 2016Playing for Data: Ground Truth from Computer Gamescited by 818
  10. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  12. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 723
  13. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 657
  14. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  15. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  16. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  17. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  18. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  19. 2016Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Agecited by 411
  20. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 349
  21. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  22. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 208
  23. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  24. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 165
  25. 2017IQA: Visual Question Answering in Interactive Environmentscited by 155
  26. 2023Vision-Language Models as Success Detectorscited by 142
  27. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 124
  28. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 118
  29. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  30. 2022Navigating to Objects in the Real Worldcited by 106
  31. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 84
  32. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  33. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Modelscited by 58
  34. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  35. 2020Scene Graph Reasoning for Visual Question Answeringcited by 50
  36. 2021PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D Worldcited by 47
  37. 2023Towards In-context Scene Understandingcited by 47
  38. 2019ShapeMask: Learning to Segment Novel Objects by Refining Shape Priorscited by 43

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 97; 2025: 262; 2026: 347.

706 papers.

Sort:

By area:

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  2. 2026World Action Models are Zero-shot Policiescited by 465
  3. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  4. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 346
  5. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  6. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  7. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 289
  8. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 251
  9. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  10. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 211
  11. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  12. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 183
38 more
  1. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 176
  2. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 166
  3. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 152
  4. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 148
  5. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 139
  6. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 138
  7. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 138
  8. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  9. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 136
  10. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 136
  11. 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 125
  12. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 120
  13. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Modelscited by 118
  14. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  15. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 114
  16. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 114
  17. 2024TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videoscited by 106
  18. 2025UP-VLA: A Unified Understanding and Prediction Model for Embodied Agentcited by 102
  19. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 96
  20. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areascited by 95
  21. 2025TesserAct: Learning 4D Embodied World Modelscited by 94
  22. 2024Vision Language Models are In-Context Value Learnerscited by 90
  23. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 89
  24. 2025Visual Spatial Tuningcited by 89
  25. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 82
  26. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 79
  27. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 75
  28. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  29. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMscited by 74
  30. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 74
  31. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 73
  32. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  33. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Modelscited by 71
  34. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 71
  35. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 69
  36. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoningcited by 68
  37. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 65
  38. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 63

50 most cited of 706 All 706 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data
  2. 2026SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models
  3. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models
  4. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  5. 2025Enhancing Spatial Reasoning through Visual and Textual Thinking
  6. 2026SpatialChain: A Benchmark for Auditing Spatial Reasoning Faithfulness in VLMs
  7. 2025SVQA-R1: Reinforcing Spatial Reasoning in MLLMs via View-Consistent Reward Optimization
  8. 2025Open3D-VQA: A Benchmark for Comprehensive Spatial Reasoning with Multimodal Large Language Model in Open Space