mlsanityData to 7 Oct 2026

RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics

arXiv 2411.16537 ↗First posted 25 Nov 2024cs.CV cs.AI cs.CLcited by 133

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Spatial understanding is a crucial capability that enables robots to perceive their surroundings, reason about their environment, and interact with it meaningfully. In modern robotics, these capabilities are increasingly provided by vision-language models. However, these models face significant challenges in spatial reasoning tasks, as their training data are based on general-purpose image datasets that often lack sophisticated spatial understanding. For example, datasets frequently do not capture reference frame comprehension, yet effective spatial reasoning requires understanding whether to reason from ego-, world-, or object-centric perspectives. To address this issue, we introduce RoboSpatial, a large-scale dataset for spatial understanding in robotics. It consists of real indoor and tabletop scenes, captured as 3D scans and egocentric images, and annotated with rich spatial information relevant to robotics. The dataset includes 1M images, 5k 3D scans, and 3M annotated spatial relationships, and the pairing of 2D egocentric images with 3D scans makes it both 2D- and 3D- ready. Our experiments show that models trained with RoboSpatial outperform baselines on downstream tasks such as spatial affordance prediction, spatial relationship prediction, and robot manipulation.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 63 of its 75 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

63 of 75 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models9 Nov
  2. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities22 Oct
  3. 2024Multi-modal Situated Reasoning in 3D Scenes4 Sep
  4. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models21 Jun
  5. 2024SpatialBot: Precise Spatial Understanding with Vision Language Models19 Jun
  6. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
  7. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models9 Jun
  8. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  9. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs12 Feb
  10. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  11. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration20 Nov
  12. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct
2 more
  1. 2023Towards Grounded Visual Spatial Reasoning in Multi-Modal Vision Language Models18 Aug
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 49

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  5. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  6. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
37 more
  1. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  6. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  7. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  8. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  9. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  10. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  11. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  12. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  13. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  14. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  15. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  16. 2022Visual Spatial Reasoningcited by 288
  17. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 187
  18. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  19. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  20. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  21. 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 118
  22. 2024CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Modelscited by 97
  23. 20226-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmarkcited by 85
  24. 2023Evaluating Spatial Understanding of Large Language Modelscited by 65
  25. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 48
  26. 2023cuRobo: Parallelized Collision-Free Minimum-Jerk Robot Motion Generationcited by 46
  27. 2023GraspGPT: Leveraging Semantic Knowledge from a Large Language Model for Task-Oriented Graspingcited by 40
  28. 2024Situational Awareness Matters in 3D Vision Language Reasoningcited by 32
  29. 2022One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestonescited by 26
  30. 2023MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognitioncited by 26
  31. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 26
  32. 2024AlanaVLM: A Multimodal Embodied AI Foundation Model for Egocentric Video Understandingcited by 22
  33. 2019NLVR2 Visual Bias Analysiscited by 19
  34. 2024Space3D-Bench: Spatial 3D Question Answering Benchmarkcited by 17
  35. 2022CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanationscited by 16
  36. 2024GRS: Generating Robotic Simulation Tasks from Real-World Imagescited by 13
  37. 2024Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Studycited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 56; 2026: 77.

133 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  3. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130
  4. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  5. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Modelscited by 110
  6. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  7. 2025Grounded Reinforcement Learning for Visual Reasoningcited by 80
  8. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  9. 2025SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planningcited by 62
  10. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 62
  11. 2025Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Modelscited by 52
  12. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 45
38 more
  1. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  2. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Reviewcited by 38
  3. 2025Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learningcited by 38
  4. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 36
  5. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  6. 20253D Aware Region Prompted Vision Language Modelcited by 31
  7. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  8. 20253D CAVLA: Leveraging Depth and 3D Context to Generalize Vision Language Action Models for Unseen Taskscited by 30
  9. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  10. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  11. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Modelscited by 24
  12. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  13. 2025SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligencecited by 19
  14. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenescited by 19
  15. 2025SpaceVista: All-Scale Visual Spatial Reasoning from mm to kmcited by 18
  16. 2026HY-Embodied-0.5: Embodied Foundation Models for Real-World Agentscited by 18
  17. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  18. 2026VideoSeek: Long-Horizon Video Agent with Tool-Guided Seekingcited by 14
  19. 2026SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoningcited by 13
  20. 2026pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoningcited by 12
  21. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answeringcited by 11
  22. 2025NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Taskscited by 10
  23. 2025Pelican-VL 1.0: A Foundation Brain Model for Embodied Intelligencecited by 10
  24. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sightcited by 8
  25. 2025SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenescited by 7
  26. 2025ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraintscited by 7
  27. 2026Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptionscited by 7
  28. 2026Thinking with Spatial Code for Physical-World Video Reasoningcited by 7
  29. 2026Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agentscited by 7
  30. 2026Hy-Embodied-VLM-1.0: Efficient Physical-World Agentscited by 7
  31. 2026PokeVLA: Empowering Pocket-Sized Vision-Language-Action Model with Comprehensive World Knowledge Guidancecited by 6
  32. 2026GEM: Generative Supervision Helps Embodied Intelligencecited by 6
  33. 2025ExtrinSplat: Decoupling Geometry and Semantics for Open-Vocabulary Understanding in 3D Gaussian Splattingcited by 5
  34. 2025IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigationcited by 5
  35. 2025Vision-Language Memory for Spatial Reasoningcited by 5
  36. 2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Modelscited by 5
  37. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 4
  38. 2025GAE: Unleashing Physical Potential of VLM with Generalizable Action Expertcited by 4

50 most cited of 133 All 133 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Technical Report of RoboSpatial Challenge at CVPR 2026: Selective Reasoning Activation and Reference-Frame Disambiguation for Embodied Spatial Reasoning
  2. 2025Embodied Spatial Intelligence: from Implicit Scene Modeling to Spatial Reasoning
  3. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks
  4. 2025RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics
  5. 2025SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes
  6. 2026ESPIRE: A Diagnostic Benchmark for Embodied Spatial Reasoning of Vision-Language Models
  7. 2026RoBoSR: Structured Scene Representations for Embodied Robotic Reasoning ↗
  8. 2026Chain of Spatial Thoughts: Modality-Agnostic Spatial Grounding for Vision Language Models