mlsanityData to 7 Oct 2026

SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models

arXiv 2406.01584 ↗First posted 3 Jun 2024cs.CVcited by 297

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Vision Language Models (VLMs) have demonstrated remarkable performance in 2D vision and language tasks. However, their ability to reason about spatial arrangements remains limited. In this work, we introduce Spatial Region GPT (SpatialRGPT) to enhance VLMs' spatial perception and reasoning capabilities. SpatialRGPT advances VLMs' spatial understanding through two key innovations: (1) a data curation pipeline that enables effective learning of regional representation from 3D scene graphs, and (2) a flexible plugin module for integrating depth information into the visual encoder of existing VLMs. During inference, when provided with user-specified region proposals, SpatialRGPT can accurately perceive their relative directions and distances. Additionally, we propose SpatialRGBT-Bench, a benchmark with ground-truth 3D annotations encompassing indoor, outdoor, and simulated environments, for evaluating 3D spatial cognition in VLMs. Our results demonstrate that SpatialRGPT significantly enhances performance in spatial reasoning tasks, both with and without local region prompts. The model also exhibits strong generalization capabilities, effectively reasoning about complex spatial relations and functioning as a region-aware dense reward annotator for robotic tasks. Code, dataset, and benchmark are released at https://www.anjiecheng.me/SpatialRGPT

Key facts. Cited by 297 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 75 of its 99 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

75 of 99 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs12 Feb
  2. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  3. 20233D Concept Learning and Reasoning from Multi-View Images20 Mar

Other references · 72

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  4. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  6. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  8. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  10. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  11. 2024Yi: Open Foundation Models by 01.AIcited by 706
  12. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  13. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  14. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  15. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  16. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  17. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
  18. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  19. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  20. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  21. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  22. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  23. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
  24. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  25. 2024InternLM2 Technical Reportcited by 391
  26. 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 389
  27. 2016Visual Dialogcited by 377
  28. 2023Segment Anything in High Qualitycited by 373
  29. 2022Visual Language Maps for Robot Navigationcited by 362
  30. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  31. 2019Scene Text Visual Question Answeringcited by 327
  32. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  33. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  34. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  35. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  36. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  37. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 258
  38. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 140; 2026: 131.

297 papers.

Sort:

By area:

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  2. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  3. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  4. 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thoughtcited by 204
  5. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  6. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  7. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 154
  8. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  9. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  10. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  11. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130
  12. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
38 more
  1. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 111
  2. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  3. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Modelscited by 110
  4. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  5. 2024MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequencecited by 85
  6. 2025Visual Spatial Tuningcited by 85
  7. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 75
  8. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  9. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoningcited by 65
  10. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 63
  11. 2025SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planningcited by 62
  12. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 62
  13. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 58
  14. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Modelscited by 54
  15. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 49
  16. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  17. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  18. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 45
  19. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 43
  20. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  21. 2025DICEPTION: A Generalist Diffusion Model for Visual Perceptual Taskscited by 37
  22. 2025Open3D-VQA: A Benchmark for Comprehensive Spatial Reasoning with Multimodal Large Language Model in Open Spacecited by 37
  23. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 36
  24. 2025Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Modelscited by 35
  25. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  26. 2025Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Modelscited by 34
  27. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  28. 20253D Aware Region Prompted Vision Language Modelcited by 31
  29. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  30. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  31. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  32. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Modelscited by 29
  33. 2025DepthLM: Metric Depth From Vision Language Modelscited by 29
  34. 2024SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Modelscited by 28
  35. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  36. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 28
  37. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  38. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27

50 most cited of 297 All 297 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models
  2. 2026SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?
  3. 2026Chain of Spatial Thoughts: Modality-Agnostic Spatial Grounding for Vision Language Models
  4. 2025Imagine in Space: Exploring the Frontier of Spatial Intelligence and Reasoning Efficiency in Vision Language Models
  5. 2025SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards
  6. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
  7. 2025Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models
  8. 2024Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoning