mlsanityData to 7 Oct 2026

Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models

arXiv 2505.17015 ↗First posted 22 May 2025cs.CV cs.CLcited by 52

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Multi-modal large language models (MLLMs) have rapidly advanced in visual tasks, yet their spatial understanding remains limited to single images, leaving them ill-suited for physical-world applications that require multi-frame reasoning. In this paper, we propose a framework to equip MLLMs with multi-frame spatial understanding by integrating fundamental spatial skills, including depth perception, visual correspondence, and dynamic perception. We design a novel data pipeline and collect the MultiSPA dataset of more than 27 million samples spanning diverse 3D and 4D scenes to enable training. Alongside MultiSPA, we introduce a comprehensive benchmark that tests a wide spectrum of spatial tasks under uniform metrics. Our resulting model, Multi-SpatialMLLM, achieves significant gains over baselines and proprietary systems, demonstrating scalable and generalizable multi-frame perception. We further observe multi-task benefits and emergent spatial capabilities in challenging scenarios, and showcase how our model can serve as a multi-frame reward annotator for robotics.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 76 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

59 of 76 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  2. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics25 Nov
  3. 2024An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models9 Nov
  4. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Models15 Sep
  5. 2024SpatialBot: Precise Spatial Understanding with Vision Language Models19 Jun
  6. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models9 Jun
  7. 2024SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors18 Mar
  8. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  9. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
38 more
  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  12. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  13. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  14. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  15. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  16. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  17. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  18. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  19. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  20. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  21. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  22. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  23. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  24. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  25. 2025Continuous 3D Perception Model with Persistent Statecited by 527
  26. 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 389
  27. 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 356
  28. 2022Visual Spatial Reasoningcited by 288
  29. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  30. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 187
  31. 2023Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perceptioncited by 166
  32. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  33. 2024TAPVid-3D: A Benchmark for Tracking Any Point in 3Dcited by 69
  34. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Modelscited by 68
  35. 2024Does Spatial Cognition Emerge in Frontier Models?cited by 65
  36. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  37. 2023DriveTrack: A Benchmark for Long-Range Point Tracking in Real-World Videoscited by 24
  38. 2024Towards Foundation Models for 3D Vision: How Close Are We?cited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 29.

52 papers.

Sort:

By area:

  1. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  2. 2025Cambrian-S: Towards Spatial Supersensing in Videocited by 123
  3. 2025Visual Spatial Tuningcited by 85
  4. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 75
  5. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  6. 2025MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligencecited by 34
  7. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  8. 2025DepthLM: Metric Depth From Vision Language Modelscited by 29
  9. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 28
  10. 2025SpaceVista: All-Scale Visual Spatial Reasoning from mm to kmcited by 18
  11. 2026Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?cited by 18
  12. 2026RHO: Your Coding Agent is Secretly a Roboticistcited by 18
38 more
  1. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  2. 2025SpatialTree: How Spatial Abilities Branch Out in MLLMscited by 11
  3. 2026MLLM-4D: Towards Visual-based Spatial-Temporal Intelligencecited by 8
  4. 2025Beyond Flatlands: Unlocking Spatial Intelligence by Decoupling 3D Reasoning from Numerical Regressioncited by 7
  5. 2026SaPaVe: Towards Active Perception and Manipulation in Vision-Language-Action Models for Roboticscited by 6
  6. 2025Video-STR: Reinforcing MLLMs in Video Spatio-Temporal Reasoning with Relation Graphcited by 5
  7. 2025ReasonX: MLLM-Guided Intrinsic Image Decompositioncited by 5
  8. 2025Knowing the Answer Isn't Enough: Fixing Reasoning Path Failures in LVLMscited by 5
  9. 2025Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoningcited by 4
  10. 2025SpatialMosaic: A Multiview VLM Dataset for Partial Visibilitycited by 4
  11. 2026OpenSpatial: A Principled Data Engine for Empowering Spatial Intelligencecited by 4
  12. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  13. 2025Scaling Spatial Reasoning in MLLMs through Programmatic Data Synthesiscited by 3
  14. 2026RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Modelscited by 3
  15. 2026Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streamscited by 3
  16. 2025Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Studycited by 2
  17. 2026OnlineSI: Taming Large Language Model for Online 3D Understanding and Groundingcited by 2
  18. 2026Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spacescited by 2
  19. 2026From Correspondence to Actions: Human-Like Multi-Image Spatial Reasoning in Multi-modal Large Language Modelscited by 2
  20. 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Modelscited by 2
  21. 2026SeGPruner: Semantic-Geometric Visual Token Pruner for 3D Question Answeringcited by 2
  22. 2026Unlocking Dense Metric Depth Estimation in VLMscited by 2
  23. 2026VLM3: Vision Language Models Are Native 3D Learnerscited by 2
  24. 2026Towards Spatial Supersensing in the Wildcited by 2
  25. 2025GTR-Bench: Evaluating Geo-Temporal Reasoning in Vision-Language Modelscited by 1
  26. 20254D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillationcited by 1
  27. 20263D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understandingcited by 1
  28. 2026HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Modelscited by 1
  29. 2026From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMscited by 1
  30. 2026Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoningcited by 1
  31. 2026OmniView-Space: Reinforcing Spatial Reasoning via Multi-Perspective Spatial Mappingcited by 1
  32. 2026Learning Multi-View Spatial Reasoning from Cross-View Relationscited by 0
  33. 2026Token Warping Helps MLLMs Look from Nearby Viewpointscited by 0
  34. 2026Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?cited by 0
  35. 2026Eliciting Complex Spatial Reasoning in MLLMs through Wide-Baseline Matchingcited by 0
  36. 2026MV-STRIDE: Enabling MLLMs to Master Multi-View Spatial Reasoning via Hierarchical Capability Modelingcited by 0
  37. 2026VCN-Bench: A Video-Contextualized Navigation Benchmark for Spatial Reasoning over Prior Visual Experiencecited by 0
  38. 2026Exemplar2VQA: A Scalable Exemplar-Driven Visual Question Answering Generation Framework via Multi-Agent Codingcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecture
  2. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence
  3. 2025See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model
  4. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models
  5. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes
  6. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes
  7. 2023Enhancing the Spatial Awareness Capability of Multi-Modal Large Language Model
  8. 2026SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision