mlsanityData to 7 Oct 2026

SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors

arXiv 2403.13438 ↗First posted 18 Mar 2024cs.CVcited by 53

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Current state-of-the-art spatial reasoning-enhanced VLMs are trained to excel at spatial visual question answering (VQA). However, we believe that higher-level 3D-aware tasks, such as articulating dynamic scene changes and motion planning, require a fundamental and explicit 3D understanding beyond current spatial VQA datasets. In this work, we present SpatialPIN, a framework designed to enhance the spatial reasoning capabilities of VLMs through prompting and interacting with priors from multiple 3D foundation models in a zero-shot, training-free manner. Extensive experiments demonstrate that our spatial reasoning-imbued VLM performs well on various forms of spatial VQA and can extend to help in various downstream robotics tasks such as pick and stack and trajectory planning.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 56 of its 74 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

56 of 74 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  2. 2022Visual Spatial Reasoning30 Apr

Other references · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  12. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
38 more
  1. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  2. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  3. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  8. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  9. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  10. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  11. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  12. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  13. 2020TartanAir: A Dataset to Push the Limits of Visual SLAMcited by 494
  14. 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 490
  15. 2019BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networkscited by 487
  16. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  17. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  18. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  19. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  20. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  21. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  22. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  23. 2023MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimationcited by 173
  24. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
  25. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  26. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  27. 2022Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB Imagescited by 120
  28. 2023SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimationcited by 115
  29. 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
  30. 2024CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Modelscited by 97
  31. 2020se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domainscited by 87
  32. 2023CNOS: A Strong Baseline for CAD-based Novel Object Segmentationcited by 86
  33. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 84
  34. 2022Perspective Fields for Single Image Camera Calibrationcited by 75
  35. 2022ShAPO: Implicit Representations for Multi-Object Shape, Appearance, and Pose Optimizationcited by 70
  36. 2023OBJECT 3DIT: Language-guided 3D-aware Image Editingcited by 62
  37. 2021DIML/CVL RGB-D Dataset: 2M RGB-D Images of Natural Indoor and Outdoor Scenescited by 45
  38. 2023Prompt a Robot to Walk with Large Language Modelscited by 44

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 25; 2026: 28.

53 papers.

Sort:

By area:

  1. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Modelscited by 110
  2. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  3. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 62
  4. 2025Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Modelscited by 52
  5. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  6. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLMcited by 41
  7. 20253D Aware Region Prompted Vision Language Modelcited by 31
  8. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  9. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 30
  10. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  11. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  12. 2025SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RLcited by 20
38 more
  1. 2026CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decodingcited by 13
  2. 2025The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Frameworkcited by 9
  3. 2025Enhancing Spatial Reasoning in Vision-Language Models via Chain-of-Thought Prompting and Reinforcement Learningcited by 9
  4. 2025Free-form language-based robotic reasoning and graspingcited by 8
  5. 2025Vision-Language Models as Differentiable Semantic and Spatial Rewards for Text-to-3D Generationcited by 6
  6. 2026Allocentric Perceiver: Disentangling Allocentric Reasoning from Egocentric Visual Priors via Frame Instantiationcited by 5
  7. 2026Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representationscited by 5
  8. 2026SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generationcited by 5
  9. 2025Aligning Text, Images, and 3D Structure Token-by-Tokencited by 4
  10. 2025Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoningcited by 4
  11. 2026SpatialEvo: Self-Evolving Spatial Intelligence via Deterministic Geometric Environmentscited by 4
  12. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  13. 2025Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction-Reasoning Synergycited by 3
  14. 2026Text-Guided 6D Object Pose Rearrangement via Closed-Loop VLM Agentscited by 3
  15. 2025Reasoning Path and Latent State Analysis for Multi-view Visual Spatial Reasoning: A Cognitive Science Perspectivecited by 2
  16. 2026Lost in Space? Vision-Language Models Struggle with Relative Camera Pose Estimationcited by 2
  17. 2025Retrieval-Augmented Defense: Adaptive and Controllable Jailbreak Prevention for Large Language Modelscited by 1
  18. 2026From Pixels to Policies: Reinforcing Spatial Reasoning in Language Models for Content-Aware Layout Designcited by 1
  19. 2026Point Cloud as a Foreign Language for Multi-modal Large Language Modelcited by 1
  20. 2026Enhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoningcited by 1
  21. 2026SPACENUM: Revisiting Spatial Numerical Understanding in VLMscited by 1
  22. 2026Grounded 3D-Aware Spatial Vision-Language Modelingcited by 1
  23. 2026FurnitureVLA: Learning Long-Horizon Bimanual Furniture Assembly with Vision-Language-Action Modelcited by 1
  24. 2026SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Videocited by 1
  25. 2026Visual Credit Audit for Multimodal Spatial Reasoningcited by 1
  26. 2026SpatialCLI: Learning to Reason With Spatial Tools, Then Without Themcited by 1
  27. 2025A Multi-Modal Neuro-Symbolic Approach for Spatial Reasoning-Based Visual Grounding in Roboticscited by 0
  28. 2025An Evaluation of Interleaved Instruction Tuning on Semantic Reasoning Performance in an Audio MLLMcited by 0
  29. 2025Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understandingcited by 0
  30. 2026EmbeWebAgent: Embedding Web Agents into Any Customized UIcited by 0
  31. 2026SpatialPoint: Spatial-aware Point Prediction for Embodied Localizationcited by 0
  32. 2026Token Warping Helps MLLMs Look from Nearby Viewpointscited by 0
  33. 2026Sketch2MinSurf: Vision-Language Guided Generation of Editable Minimal Surfaces from Hand-Drawn Sketchescited by 0
  34. 2026CAIRN: Cross-Room 3D Scene Understanding with Topology-Aware Large Multimodal Modelscited by 0
  35. 2026GReFEM: Multimodal LLMs as Zero-Shot Semantic Assistants for Physics-Guided 3D Mesh Refinementcited by 0
  36. 2026When Depth Is Better Told Than Shown: Depth-Ordinal Prompting for Vision-Language Spatial Reasoningcited by 0
  37. 2026SpatialQuery: Benchmarking Geometry-Grounded Multi-Instance Spatial Reasoning in Vision-Language Modelscited by 0
  38. 2026Seeing Is Not Measuring: Tool-Augmented Metric Spatial Reasoning for Vision-Language Modelscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models
  2. 2026Distilling 3D Spatial Reasoning into a Lightweight Vision-Language Model with CoT
  3. 2026SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models
  4. 2025SVQA-R1: Reinforcing Spatial Reasoning in MLLMs via View-Consistent Reward Optimization
  5. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning
  6. 2025Enhancing Spatial Reasoning through Visual and Textual Thinking
  7. 2026SPATIOROUTE: Dynamic Prompt Routing for Zero-Shot Spatial Reasoning
  8. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models