mlsanityData to 14 Oct 2026

MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs

arXiv 2503.13111 ↗First posted 17 Mar 2025cs.CV cs.CL cs.LGcited by 73

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Multimodal large language models (MLLMs) excel at 2D visual understanding but remain limited in their ability to reason about 3D space. In this work, we leverage large-scale high-quality 3D scene data with open-set annotations to introduce 1) a novel supervised fine-tuning dataset and 2) a new evaluation benchmark, focused on indoor scenes. Our Cubify Anything VQA (CA-VQA) data covers diverse spatial tasks including spatial relationship prediction, metric size and distance estimation, and 3D grounding. We show that CA-VQA enables us to train MM-Spatial, a strong generalist MLLM that also achieves state-of-the-art performance on 3D spatial understanding benchmarks, including our own. We show how incorporating metric depth and multi-view inputs (provided in CA-VQA) can further improve 3D understanding, and demonstrate that data alone allows our model to achieve depth perception capabilities comparable to dedicated monocular depth estimation models.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 117 of its 134 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

117 of 134 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs5 Feb
  2. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  3. 2024LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences2 Dec
  4. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  5. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics25 Nov
  6. 2024An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models9 Nov
  7. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities22 Oct
  8. 2024MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA9 Oct
  9. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness26 Sep
  10. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models15 Sep
  11. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models9 Jun
  12. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
5 more
  1. 2024Language-Image Models with 3D Understanding6 May
  2. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs8 Apr
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  4. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  5. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct

Other references · 100

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  10. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
38 more
  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  4. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  6. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  7. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  8. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  10. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  11. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  12. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  13. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  14. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  15. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  16. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  17. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  18. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  19. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  20. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  21. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  22. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  23. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
  24. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  25. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  26. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 404
  27. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  28. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  29. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  30. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  31. 2024What matters when building vision-language models?cited by 326
  32. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  33. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  34. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  35. 2022Visual Spatial Reasoningcited by 293
  36. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  37. 2023Data Filtering Networkscited by 250
  38. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247

50 most cited of 100 All 100 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 30; 2026: 43.

73 papers.

Sort:

By area:

  1. 2025Visual Spatial Tuningcited by 89
  2. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 65
  3. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 63
  4. 2025Evo-0: Vision-Language-Action Model with Implicit Spatial Understandingcited by 58
  5. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 43
  6. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 33
  7. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  8. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Modelscited by 25
  9. 2025SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligencecited by 24
  10. 2025SVQA-R1: Reinforcing Spatial Reasoning in MLLMs via View-Consistent Reward Optimizationcited by 16
  11. 2025SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewardscited by 16
  12. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
38 more
  1. 2025BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Datasetcited by 12
  2. 2025From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMscited by 12
  3. 2026On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMscited by 11
  4. 2026Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Trainingcited by 11
  5. 2025Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Modelscited by 10
  6. 2025Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Drivingcited by 9
  7. 2025Beyond Flatlands: Unlocking Spatial Intelligence by Decoupling 3D Reasoning from Numerical Regressioncited by 8
  8. 2026Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptionscited by 7
  9. 2026Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluationcited by 6
  10. 2026OpenSpatial: A Principled Data Engine for Empowering Spatial Intelligencecited by 6
  11. 2025IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigationcited by 5
  12. 2025SpatialMosaic: A Multiview VLM Dataset for Partial Visibilitycited by 5
  13. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agentscited by 5
  14. 2026RieMind: Geometry-Grounded Spatial Agent for Scene Understandingcited by 4
  15. 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoningcited by 4
  16. 2026ViSRA: A Video-based Spatial Reasoning Agent for Multi-modal Large Language Modelscited by 4
  17. 2026MultiView-Bench: A Diagnostic Benchmark for World-Centric Multi-View Integration in VLMscited by 4
  18. 2025MIRAGE: A Multi-modal Benchmark for Spatial Perception, Reasoning, and Intelligencecited by 3
  19. 2025From Diagnosis to Improvement: Probing Spatio-Physical Reasoning in Vision Language Modelscited by 3
  20. 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modelingcited by 3
  21. 2026From Correspondence to Actions: Human-Like Multi-Image Spatial Reasoning in Multi-modal Large Language Modelscited by 3
  22. 2026HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Modelscited by 3
  23. 2026Are VLMs Lost Between Sky and Space? LinkS2Bench for UAV-Satellite Dynamic Cross-View Spatial Intelligencecited by 3
  24. 2026CrossView Suite: Harnessing Cross-view Spatial Intelligence of MLLMs with Dataset, Model and Benchmarkcited by 3
  25. 2026SPACENUM: Revisiting Spatial Numerical Understanding in VLMscited by 3
  26. 2026Lost in Space? Vision-Language Models Struggle with Relative Camera Pose Estimationcited by 2
  27. 2026SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMscited by 2
  28. 2026Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sportscited by 2
  29. 2026Multimodal Language Models Cannot Spot Spatial Inconsistenciescited by 2
  30. 2026Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?cited by 2
  31. 2026SATURN: Symbolic Spatial Reasoning for Multi-Perspective Groundingcited by 2
  32. 2026Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoningcited by 2
  33. 2025Human Centric General Physical Intelligence for Agile Manufacturing Automationcited by 1
  34. 2025Mitigating Cross-Image Information Leakage in Multi-Image Understanding with Large Vision-Language Modelscited by 1
  35. 2025Automated stereotactic radiosurgery planning using a human-in-the-loop reasoning large language model agentcited by 1
  36. 2025Thinking on Maps: How Foundation Model Agents Explore, Remember, and Reason Map Environmentscited by 1
  37. 2026Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVAcited by 1
  38. 2026SpaceDG: Benchmarking Spatial Intelligence under Visual Degradationcited by 1

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models
  2. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  3. 2026SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision
  4. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLM
  5. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence
  6. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors
  7. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence
  8. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations