mlsanityData to 7 Oct 2026

LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness

arXiv 2409.18125 ↗First posted 26 Sep 2024cs.CVcited by 178

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D scene understanding capabilities has been hindered by the lack of large-scale 3D vision-language datasets and powerful 3D encoders. In this paper, we introduce a simple yet effective framework called LLaVA-3D. Leveraging the strong 2D visual understanding priors from LLaVA, our LLaVA-3D efficiently adapts LLaVA for 3D scene understanding without compromising 2D understanding capabilities. To achieve this, we utilize the 3D position embeddings to enhance the 2D CLIP Patches with 3D spatial context information and construct 3D patches. By integrating the 3D position embeddings into 2D LMMs and employing joint 2D and 3D vision-language instruction tuning, we establish a unified architecture for both 2D visual understanding and 3D scene understanding. In contrast to previous 3D LMMs, LLaVA-3D supports decoding accurate 3D spatial perception outputs, e.g., 3D bounding boxes, directly from these 3D patches, without relying on the time-consuming off-the-shelf 3D segmentors. Experimental results show that LLaVA-3D converges 3.5x faster than existing 3D LMMs when trained on 3D vision-language datasets. Moreover, LLaVA-3D not only achieves state-of-the-art performance across various 3D tasks but also maintains comparable 2D visual understanding and vision-language conversation capabilities with LLaVA.

Key facts. Cited by 178 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 60 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

47 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  11. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
34 more
  1. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  5. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  6. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  7. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  8. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  9. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  10. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  11. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  12. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 351
  13. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  14. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  15. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  16. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  17. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  18. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  19. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  20. 2021Group-Free 3D Object Detection via Transformerscited by 215
  21. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  22. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  23. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  24. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  25. 2022Multi-View Transformer for 3D Visual Groundingcited by 138
  26. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  27. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  28. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  29. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  30. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 77
  31. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  32. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  33. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  34. 2024ODIN: A Single Model for 2D and 3D Segmentationcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 93; 2026: 80.

178 papers.

Sort:

By area:

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  2. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  3. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 154
  4. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  5. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  7. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  8. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  9. 2025PointVLA: Injecting the 3D World into Vision-Language-Action Modelscited by 101
  10. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Modelscited by 83
  11. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  12. 2025GeoVLA: Empowering 3D Representations in Vision-Language-Action Modelscited by 68
38 more
  1. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  2. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 63
  3. 2025Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenescited by 62
  4. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  5. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  6. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLMcited by 41
  7. 2025G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoningcited by 39
  8. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  9. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  10. 2025Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spacescited by 33
  11. 20253D Aware Region Prompted Vision Language Modelcited by 31
  12. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  13. 2025N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Modelscited by 28
  14. 2025Introducing Visual Perception Token into Multimodal Large Language Modelcited by 26
  15. 20253DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Modelcited by 25
  16. 2025SplatTalk: 3D VQA with Gaussian Splattingcited by 24
  17. 2025OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understandingcited by 24
  18. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  19. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 22
  20. 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligencecited by 22
  21. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 22
  22. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  23. 2025Emergent Outlier View Rejection in Visual Geometry Grounded Transformerscited by 19
  24. 2024DepthCues: Evaluating Monocular Depth Perception in Large Vision Modelscited by 18
  25. 2025Reasoning in Space via Grounding in the Worldcited by 18
  26. 2025LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understandingcited by 17
  27. 2025Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMscited by 16
  28. 2025Unifying 2D and 3D Vision-Language Understandingcited by 15
  29. 2025Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview imagescited by 15
  30. 2026TwinBrainVLA: Unleashing the Potential of Generalist VLMs for Embodied Tasks via Asymmetric Mixture-of-Transformerscited by 14
  31. 2025NuGrounding: A Multi-View 3D Visual Grounding Framework in Autonomous Drivingcited by 12
  32. 2025SpatialTree: How Spatial Abilities Branch Out in MLLMscited by 11
  33. 2025Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understandingcited by 10
  34. 2025Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMscited by 10
  35. 2025Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generationcited by 10
  36. 2025Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videoscited by 10
  37. 2026RayRoPE: Projective Ray Positional Encoding for Multi-view Attentioncited by 10
  38. 2025UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encodingcited by 9

50 most cited of 178 All 178 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLaVA3: Representing 3D Scenes like a Cubist Painter to Boost 3D Scene Understanding of VLMs
  2. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer
  3. 2025Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models
  4. 2026Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
  5. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  6. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding
  7. 2025Unleashing the Multi-View Fusion Potential: Noise Correction in VLM for Open-Vocabulary 3D Scene Understanding
  8. 2026Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering