mlsanityData to 7 Oct 2026

Beyond Single Expert: Harmonizing Diverse Visual Priors in MLLMs for Spatial Understanding

arXiv 2607.15054 ↗First posted 16 Jul 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated substantial promise in spatial understanding. Existing works typically incorporate prior knowledge extracted from a pre-trained foundation model to further enhance the spatial awareness of MLLMs. In this paper, we first reveal that when integrating diverse foundation models into MLLMs, different models provide complementary spatial priors that benefit different tasks. Motivated by this, we propose ViPS, a novel multi-model prior framework designed to fully unleash the potential of incorporating multiple Visual Priors from diverse models into MLLMs for Spatial understanding. Specifically, ViPS introduces an Efficient Prior Proxy to generate multiple foundational priors with minimal inference overhead, and a Dynamic Prior Fusion mechanism to achieve harmonious and context-aware prior fusion and injection from the prior proxies. Extensive experiments demonstrate that ViPS successfully harmonizes diverse visual priors, establishing new state-of-the-art performance across multiple complex spatial reasoning and 3D spatial understanding benchmarks. Project page: https://visual-ai.github.io/vips

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 56 references that are in the map. First posted on arXiv 16 Jul 2026.

Builds on

50 of 56 references.

Closest ancestors · 19

Same direction, newest first

  1. 2026SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning28 Mar
  2. 2026Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding19 Mar
  3. 2026Thinking with Geometry: Active Geometry Integration for Spatial Reasoning5 Feb
  4. 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modeling1 Dec
  5. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding2 Jun
  6. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors30 May
  7. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction26 May
  8. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness2 Apr
  9. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoning2 Apr
  10. 2025Improved Visual-Spatial Reasoning via R1-Zero-Like Training1 Apr
  11. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D29 Mar
  12. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning1 Mar
7 more
  1. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models2 Jan
  2. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer2 Jan
  3. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  4. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  5. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors2 May
  6. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generation5 Dec
  7. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  6. 2024GPT-4o System Cardcited by 4,467
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  11. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
19 more
  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  3. 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 626
  4. 2024Long Context Transfer from Language to Visioncited by 502
  5. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  6. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  7. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  8. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  9. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  10. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  11. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  12. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  13. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  14. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  15. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  16. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  17. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 88
  18. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  19. 2025Trace Anything: Representing Any Video in 4D via Trajectory Fieldscited by 31

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence
  2. 2026SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision
  3. 2025Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models
  4. 2025Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecture
  5. 2026GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models
  6. 2026LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models
  7. 2025Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes
  8. 2026MV-STRIDE: Enabling MLLMs to Master Multi-View Spatial Reasoning via Hierarchical Capability Modeling