mlsanityData to 7 Oct 2026

Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence

arXiv 2505.23747 ↗First posted 29 May 2025cs.CV cs.AI cs.LGcited by 206

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recent advancements in Multimodal Large Language Models (MLLMs) have significantly enhanced performance on 2D visual tasks. However, improving their spatial intelligence remains a challenge. Existing 3D MLLMs always rely on additional 3D or 2.5D data to incorporate spatial awareness, restricting their utility in scenarios with only 2D inputs, such as images or videos. In this paper, we present Spatial-MLLM, a novel framework for visual-based spatial reasoning from purely 2D observations. Unlike conventional video MLLMs which rely on CLIP-based visual encoders optimized for semantic understanding, our key insight is to unleash the strong structure prior from the feed-forward visual geometry foundation model. Specifically, we propose a dual-encoder architecture: a pretrained 2D visual encoder to extract semantic features, and a 3D spatial encoder-initialized from the backbone of the visual geometry model-to extract 3D structure features. A connector then integrates both features into unified visual tokens for enhanced spatial understanding. Furthermore, we propose a space-aware frame sampling strategy at inference time, which selects the spatially informative frames of a video sequence, ensuring that even under limited token length, the model focuses on frames critical for spatial reasoning. Beyond architecture improvements, we construct a training dataset from multiple sources and train the model on it using supervised fine-tuning and GRPO. Extensive experiments on various real-world datasets demonstrate that Spatial-MLLM achieves state-of-the-art performance in a wide range of visual-based spatial understanding and reasoning tasks. Project page: https://diankun-wu.github.io/Spatial-MLLM/.

Key facts. Cited by 206 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 75 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

62 of 75 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction26 May
  2. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoning2 Apr
  3. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?31 Mar
  4. 2025Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Models21 Mar
  5. 2025ST-Think: How Multimodal Large Language Models Reason About 4D Worlds from Ego-Centric Videos16 Mar
  6. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models2 Jan
  7. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer2 Jan
  8. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  9. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  10. 2024Multi-modal Situated Reasoning in 3D Scenes4 Sep
  11. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  12. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov

Other references · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2025Qwen2.5-VL Technical Reportcited by 5,418
  9. 2024GPT-4o System Cardcited by 4,467
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
38 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  2. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  7. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  9. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  10. 2023Vision Transformers Need Registerscited by 844
  11. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  12. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  13. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  14. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  15. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  16. 2024Long Context Transfer from Language to Visioncited by 502
  17. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  18. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
  19. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  20. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  21. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  22. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  23. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  24. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  25. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  26. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  27. 2025Transfer between Modalities with MetaQueriescited by 231
  28. 2024MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videoscited by 224
  29. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  30. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  31. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  32. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  33. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  34. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  35. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  36. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  37. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  38. 2025TULIP: Towards Unified Language-Image Pretrainingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 57; 2026: 149.

206 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 86
  3. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Modelscited by 83
  4. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 80
  5. 2025Scaling Spatial Intelligence with Multimodal Foundation Modelscited by 75
  6. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 58
  7. 2025Evo-0: Vision-Language-Action Model with Implicit Spatial Understandingcited by 55
  8. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  9. 2025G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoningcited by 39
  10. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  11. 2025MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligencecited by 34
  12. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
38 more
  1. 2025Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecturecited by 30
  2. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 28
  3. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  4. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  5. 2025From Spatial to Actions: Grounding Vision-Language-Action Model in Spatial Foundation Priorscited by 25
  6. 2025DiT360: High-Fidelity Panoramic Image Generation via Hybrid Trainingcited by 24
  7. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 22
  8. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 22
  9. 2026Thinking with Blueprints: Assisting Vision-Language Models in Spatial Reasoning via Structured Object Representationcited by 21
  10. 2026ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loopcited by 19
  11. 2025SpaceVista: All-Scale Visual Spatial Reasoning from mm to kmcited by 18
  12. 2025Reasoning in Space via Grounding in the Worldcited by 18
  13. 2026Vision-aligned Latent Reasoning for Multi-modal Large Language Modelcited by 16
  14. 2025Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview imagescited by 15
  15. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planningcited by 15
  16. 2026Thinking with Geometry: Active Geometry Integration for Spatial Reasoningcited by 14
  17. 2025Towards Spatial Trace with Reasoning in Vision-Language Models for Roboticscited by 12
  18. 2026pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoningcited by 12
  19. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  20. 2025Euclid's Gift: Enhancing Spatial Perception and Reasoning in Vision-Language Models via Geometric Surrogate Taskscited by 11
  21. 2026Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D Worldcited by 11
  22. 2026ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoningcited by 11
  23. 2025Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesiscited by 10
  24. 2025Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Modelscited by 10
  25. 2025SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagerycited by 10
  26. 2026MetricAnything: Scaling Metric Depth Pretraining with Noisy Heterogeneous Sourcescited by 10
  27. 2026PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarioscited by 10
  28. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  29. 2025Position: Reasoning After Perception Means Reasoning Without Visioncited by 8
  30. 2026Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Mapscited by 8
  31. 2026MLLM-4D: Towards Visual-based Spatial-Temporal Intelligencecited by 8
  32. 2026Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Trainingcited by 8
  33. 2025Generalizable Operating Room Expert with Multimodal Enhancementcited by 7
  34. 2025Beyond Flatlands: Unlocking Spatial Intelligence by Decoupling 3D Reasoning from Numerical Regressioncited by 7
  35. 2026Can LLMs See Without Pixels? Benchmarking Spatial Intelligence from Textual Descriptionscited by 7
  36. 2026CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoningcited by 7
  37. 2026Thinking with Spatial Code for Physical-World Video Reasoningcited by 7
  38. 2026S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligencecited by 7

50 most cited of 206 All 206 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision
  2. 2026Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations
  3. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes
  4. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models
  5. 2025SpatialGeo:Boosting Spatial Reasoning in Multimodal LLMs via Geometry-Semantics Fusion
  6. 2025SpatialMosaic: A Multiview VLM Dataset for Partial Visibility
  7. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors
  8. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D