mlsanityData to 7 Oct 2026

Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding

arXiv 2412.00493 ↗First posted 30 Nov 2024cs.CV cs.CLcited by 131

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

The rapid advancement of Multimodal Large Language Models (MLLMs) has significantly impacted various multimodal tasks. However, these models face challenges in tasks that require spatial understanding within 3D environments. Efforts to enhance MLLMs, such as incorporating point cloud features, have been made, yet a considerable gap remains between the models' learned representations and the inherent complexity of 3D scenes. This discrepancy largely stems from the training of MLLMs on predominantly 2D data, which restricts their effectiveness in comprehending 3D spaces. To address this issue, in this paper, we propose a novel generalist model, i.e., Video-3D LLM, for 3D scene understanding. By treating 3D scenes as dynamic videos and incorporating 3D position encoding into these representations, our Video-3D LLM aligns video representations with real-world spatial contexts more accurately. In addition, we have implemented a maximum coverage sampling technique to optimize the trade-off between computational cost and performance. Extensive experiments demonstrate that our model achieves state-of-the-art performance on several 3D scene understanding benchmarks, including ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.

Key facts. Cited by 131 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 56 references that are in the map. First posted on arXiv 30 Nov 2024.

Builds on

44 of 56 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness26 Sep
  2. 2024Multi-modal Situated Reasoning in 3D Scenes4 Sep
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  4. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  10. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
27 more
  1. 2024Qwen2 Technical Reportcited by 2,076
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  3. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  5. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  6. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  7. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  8. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  9. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  10. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  11. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  12. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  13. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  14. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  15. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  16. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 187
  17. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  18. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  19. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  20. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  21. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  22. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 125
  23. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  24. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  25. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  26. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  27. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 55; 2026: 76.

131 papers.

Sort:

By area:

  1. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  2. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 154
  3. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  4. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  5. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Modelscited by 110
  6. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  7. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Modelscited by 83
  8. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  9. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  10. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 63
  11. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  12. 2025G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoningcited by 39
38 more
  1. 20253D Aware Region Prompted Vision Language Modelcited by 31
  2. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  3. 2025SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Drivingcited by 28
  4. 2025N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Modelscited by 28
  5. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  6. 2025Reasoning in Space via Grounding in the Worldcited by 18
  7. 2026Utonia: Toward One Encoder for All Point Cloudscited by 18
  8. 2025LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understandingcited by 17
  9. 2025Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview imagescited by 15
  10. 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 14
  11. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Groundingcited by 12
  12. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 11
  13. 2025Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generationcited by 10
  14. 2025UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encodingcited by 9
  15. 2025Towards Scalable Spatial Intelligence via 2D-to-3D Data Liftingcited by 8
  16. 2026CoV: Chain-of-View Prompting for Spatial Reasoningcited by 8
  17. 2026Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Mapscited by 8
  18. 2026CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelingcited by 8
  19. 2025SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenescited by 7
  20. 2025Beyond Flatlands: Unlocking Spatial Intelligence by Decoupling 3D Reasoning from Numerical Regressioncited by 7
  21. 2025Part-X-MLLM: Part-aware 3D Multimodal Large Language Modelcited by 6
  22. 2026Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understandingcited by 6
  23. 2026Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligencecited by 6
  24. 2025Spatio-Temporal LLM: Reasoning about Environments and Actionscited by 5
  25. 2025NAUTILUS: A Large Multimodal Model for Underwater Scene Understandingcited by 5
  26. 2025Vision-Language Memory for Spatial Reasoningcited by 5
  27. 2025CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoningcited by 5
  28. 2025D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigationcited by 5
  29. 2026Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Modelscited by 5
  30. 2026Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representationscited by 5
  31. 2026Do 3D Large Language Models Really Understand 3D Spatial Relationships?cited by 5
  32. 2026PanoWorld: Towards Spatial Supersensing in 360^circ Panorama Worldcited by 5
  33. 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Expertscited by 4
  34. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 4
  35. 2025Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoningcited by 4
  36. 2025Zoo3D: Zero-Shot 3D Object Detection at Scene Levelcited by 4
  37. 2026Make Geometry Matter for Spatial Reasoningcited by 4
  38. 2025Warehouse Spatial Question Answering with LLM Agentcited by 3

50 most cited of 131 All 131 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  2. 2025Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models
  3. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLM
  4. 2025HMR3D: Hierarchical Multimodal Representation for 3D Scene Understanding with Large Vision-Language Model
  5. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding
  6. 2024SceneGPT: A Language Model for 3D Scene Understanding
  7. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
  8. 20233D-LLM: Injecting the 3D World into Large Language Models