mlsanityData to 14 Oct 2026

SpatialBot: Precise Spatial Understanding with Vision Language Models

arXiv 2406.13642 ↗First posted 19 Jun 2024cs.CVcited by 152

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision Language Models (VLMs) have achieved impressive performance in 2D image understanding, however they are still struggling with spatial understanding which is the foundation of Embodied AI. In this paper, we propose SpatialBot for better spatial understanding by feeding both RGB and depth images. Additionally, we have constructed the SpatialQA dataset, which involves multi-level depth-related questions to train VLMs for depth understanding. Finally, we present SpatialBench to comprehensively evaluate VLMs' capabilities in spatial understanding at different levels. Extensive experiments on our spatial-understanding benchmark, general VLM benchmarks and Embodied AI tasks, demonstrate the remarkable improvements of SpatialBot trained on SpatialQA. The model, code and data are available at https://github.com/BAAI-DCAI/SpatialBot.

Key facts. Cited by 152 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 74 of its 97 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

74 of 97 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  2. 2024Language-Image Models with 3D Understanding6 May
  3. 2024Proximity QA: Unleashing the Power of Multi-Modal Large Language Models for Spatial Proximity Analysis31 Jan
  4. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  5. 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation24 Dec
  6. 2023Physically Grounded Vision-Language Models for Robotic Manipulation5 Sep
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 67

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Segment Anythingcited by 8,234
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  12. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  10. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  11. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  12. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
  13. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  14. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  15. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  16. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  17. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  18. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 723
  19. 2024Yi: Open Foundation Models by 01.AIcited by 719
  20. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  21. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 657
  22. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  23. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  24. 2023Segment Everything Everywhere All at Oncecited by 548
  25. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  26. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  27. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  28. 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 406
  29. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  30. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  31. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
  32. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 327
  33. 2017Sparsity Invariant CNNscited by 326
  34. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  35. 2017Spatial As Deep: Spatial CNN for Traffic Scene Understandingcited by 187
  36. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  37. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  38. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 83; 2026: 56.

152 papers.

Sort:

By area:

  1. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 136
  2. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 120
  3. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 114
  4. 2024MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequencecited by 89
  5. 2025Visual Spatial Tuningcited by 89
  6. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 79
  7. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoningcited by 68
  8. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 65
  9. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 62
  10. 2025Evo-0: Vision-Language-Action Model with Implicit Spatial Understandingcited by 58
  11. 2024Reconstructive Visual Instruction Tuningcited by 55
  12. 2025Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Modelscited by 52
38 more
  1. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  2. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 46
  3. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  4. 2025RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correctioncited by 41
  5. 2025Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learningcited by 41
  6. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 39
  7. 2024Core Knowledge Deficits in Multi-Modal Language Modelscited by 38
  8. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 35
  9. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  10. 20253D Aware Region Prompted Vision Language Modelcited by 33
  11. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 33
  12. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 32
  13. 2025DepthLM: Metric Depth From Vision Language Modelscited by 30
  14. 2024SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Modelscited by 29
  15. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 29
  16. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  17. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  18. 2026ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loopcited by 26
  19. 2025SIFThinker: Spatially-Aware Image Focus for Visual Reasoningcited by 25
  20. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Modelscited by 25
  21. 2025SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligencecited by 24
  22. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
  23. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 24
  24. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipescited by 22
  25. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?cited by 21
  26. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenescited by 20
  27. 2025Reasoning in Space via Grounding in the Worldcited by 19
  28. 2025PhysVLM: Enabling Visual Language Models to Understand Robotic Physical Reachabilitycited by 18
  29. 2025SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewardscited by 16
  30. 2026Everything in Its Place: Benchmarking Spatial Intelligence of Text-to-Image Modelscited by 16
  31. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 13
  32. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 12
  33. 2025Memory-Centric Embodied Question Answeringcited by 12
  34. 2025Depth Anything at Any Conditioncited by 12
  35. 2025BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Datasetcited by 12
  36. 2025From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMscited by 12
  37. 2024Graph Canvas for Controllable 3D Scene Generationcited by 11
  38. 2025ToSA: Token Merging with Spatial Awarenesscited by 11

50 most cited of 152 All 152 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The Spatial Blindspot of Vision-Language Models
  2. 2025SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data
  3. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models
  4. 2026Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models
  5. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D
  6. 2026Chain of Spatial Thoughts: Modality-Agnostic Spatial Grounding for Vision Language Models
  7. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models
  8. 2025Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecture