mlsanityData to 14 Oct 2026

3D Aware Region Prompted Vision Language Model

arXiv 2509.13317 ↗First posted 16 Sep 2025cs.CVcited by 33

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

We present Spatial Region 3D (SR-3D) aware vision-language model that connects single-view 2D images and multi-view 3D data through a shared visual token space. SR-3D supports flexible region prompting, allowing users to annotate regions with bounding boxes, segmentation masks on any frame, or directly in 3D, without the need for exhaustive multi-frame labeling. We achieve this by enriching 2D visual features with 3D positional embeddings, which allows the 3D model to draw upon strong 2D priors for more accurate spatial reasoning across frames, even when objects of interest do not co-occur within the same view. Extensive experiments on both general 2D vision language and specialized 3D spatial benchmarks demonstrate that SR-3D achieves state-of-the-art performance, underscoring its effectiveness for unifying 2D and 3D representation space on scene understanding. Moreover, we observe applicability to in-the-wild videos without sensory 3D inputs or ground-truth 3D annotations, where SR-3D accurately infers spatial relationships and metric measurements.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 93 of its 104 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

93 of 104 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  2. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models10 Dec
  3. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  4. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics25 Nov
  5. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness26 Sep
  6. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models15 Sep
  7. 2024Multi-modal Situated Reasoning in 3D Scenes4 Sep
  8. 2024SpatialBot: Precise Spatial Understanding with Vision Language Models19 Jun
  9. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
  10. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  11. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors2 May
  12. 2024SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors18 Mar
2 more
  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  2. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov

Other references · 79

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
38 more
  1. 2019Towards VQA Models That Can Readcited by 1,718
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  3. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
  4. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  5. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  6. 2024Depth Anything V2cited by 1,422
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  10. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  11. 2024Grounding Image Matching in 3D with MASt3Rcited by 983
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  13. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  14. 2016A Diagram Is Worth A Dozen Imagescited by 804
  15. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  16. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  17. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  18. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  19. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  20. 2025Continuous 3D Perception Model with Persistent Statecited by 548
  21. 2024Long Context Transfer from Language to Visioncited by 515
  22. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  23. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  24. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  25. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
  26. 2024InternLM2 Technical Reportcited by 397
  27. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  28. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  29. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  30. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  31. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  32. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  33. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  34. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  35. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  36. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  37. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 211
  38. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 25.

33 papers.

Sort:

By area:

  1. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  2. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 23
  3. 2026Thinking with Blueprints: Assisting Vision-Language Models in Spatial Reasoning via Structured Object Representationcited by 21
  4. 2025VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMscited by 13
  5. 2025Towards Spatial Trace with Reasoning in Vision-Language Models for Roboticscited by 12
  6. 2025From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMscited by 12
  7. 2026Make Geometry Matter for Spatial Reasoningcited by 8
  8. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  9. 2025POMA-3D: The Point Map Way to 3D Scene Understandingcited by 4
  10. 20263D-Layout-R1: Structured Reasoning for Language-Instructed Spatial Editingcited by 4
  11. 2025LAST: LeArning to Think in Space and Time for Generalist Vision-Language Modelscited by 3
  12. 2026Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Mapscited by 3
21 more
  1. 2026HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Modelscited by 3
  2. 2026SeGPruner: Semantic-Geometric Visual Token Pruner for 3D Question Answeringcited by 3
  3. 2026Grounded 3D-Aware Spatial Vision-Language Modelingcited by 3
  4. 20254D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillationcited by 2
  5. 2026What if? Emulative Simulation with World Models for Situated Reasoningcited by 2
  6. 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Modelscited by 2
  7. 2026Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignmentcited by 2
  8. 20263D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understandingcited by 1
  9. 20264DP-QA: Scalable QA for 4D Perception in Vision Language Modelscited by 1
  10. 2026Reinforcing Dual-Path Reasoning in Spatial Vision Language Modelscited by 1
  11. 2026CoverPrune: Coverage-Driven Token Pruning for 3D VLMs via Optimal Transportcited by 1
  12. 2026BEACON: Language-Conditioned Navigation Affordance Prediction under Occlusioncited by 0
  13. 2026Token Warping Helps MLLMs Look from Nearby Viewpointscited by 0
  14. 2026Foresight: Iterative Reasoning About Clues that Matter for Navigationcited by 0
  15. 2026OneCanvas: 3D Scene Understanding via Panoramic Reprojectioncited by 0
  16. 2026Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understandingcited by 0
  17. 2026OrientSAM: Mitigating Camera-Centric Shortcut in Multimodal Spatial Reasoning via Orientation-Aware Spatial Alignmentcited by 0
  18. 20263DZip: Spatial-Aware Feature Diversity-Guided Token Compression for 3D Question Answeringcited by 0
  19. 2026Metric-Bench: Exploring In-context Spatial Metric Reasoning in VLMs for Indoor Scenescited by 0
  20. 2026From Sight to Foresight: Predictive Spatial Reasoning in Vision-Language Modelscited by 0
  21. 2026GeoReader: Learning Latent Geometry for Spatial Reasoning in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models
  2. 2025Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models
  3. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D
  4. 2026Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
  5. 20253D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation
  6. 2026SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision
  7. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Data
  8. 2026Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoning