mlsanityData to 14 Oct 2026

3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer

arXiv 2501.01163 ↗First posted 2 Jan 2025cs.CVcited by 68

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Current 3D Large Multimodal Models (3D LMMs) have shown tremendous potential in 3D-vision-based dialogue and reasoning. However, how to further enhance 3D LMMs to achieve fine-grained scene understanding and facilitate flexible human-agent interaction remains a challenging problem. In this work, we introduce 3D-LLaVA, a simple yet highly powerful 3D LMM designed to act as an intelligent assistant in comprehending, reasoning, and interacting with the 3D world. Unlike existing top-performing methods that rely on complicated pipelines-such as offline multi-view feature extraction or additional task-specific heads-3D-LLaVA adopts a minimalist design with integrated architecture and only takes point clouds as input. At the core of 3D-LLaVA is a new Omni Superpoint Transformer (OST), which integrates three functionalities: (1) a visual feature selector that converts and selects visual tokens, (2) a visual prompt encoder that embeds interactive visual prompts into the visual token space, and (3) a referring mask decoder that produces 3D masks based on text description. This versatile OST is empowered by the hybrid pretraining to obtain perception priors and leveraged as the visual connector that bridges the 3D data to the LLM. After performing unified instruction tuning, our 3D-LLaVA reports impressive results on various benchmarks.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 70 references that are in the map. First posted on arXiv 2 Jan 2025.

Builds on

54 of 70 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model27 May
  2. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
  3. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes12 Apr

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  3. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  4. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  5. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  6. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 385
  7. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  8. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  9. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  10. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 328
  11. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  12. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
  13. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  14. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  15. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  16. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  17. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  18. 2023SparseBEV: High-Performance Sparse 3D Object Detection from Multi-Camera Videoscited by 150
  19. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  20. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  21. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  22. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  23. 2022Superpoint Transformer for 3D Scene Instance Segmentationcited by 116
  24. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  25. 2023OneFormer3D: One Transformer for Unified Point Cloud Segmentationcited by 95
  26. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 78
  27. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  28. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  29. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  30. 2023Mask-Attention-Free Transformer for 3D Instance Segmentationcited by 46
  31. 2022Spatiality-guided Transformer for 3D Dense Captioning on Point Cloudscited by 40
  32. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  33. 2023Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioningcited by 39
  34. 20233D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentationcited by 36
  35. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 35
  36. 2023Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Groundingcited by 32
  37. 2024Agent3D-Zero: An Agent for Zero-shot 3D Understandingcited by 29
  38. 2022Contextual Modeling for 3D Dense Captioning on Point Cloudscited by 18

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 31; 2026: 37.

68 papers.

Sort:

By area:

  1. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  2. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 166
  3. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 70
  4. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLMcited by 42
  5. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 33
  6. 2025Multimodal Referring Segmentation: A Surveycited by 22
  7. 2025URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Modelcited by 21
  8. 2025Reasoning in Space via Grounding in the Worldcited by 19
  9. 2025LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understandingcited by 17
  10. 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 15
  11. 2025NuGrounding: A Multi-View 3D Visual Grounding Framework in Autonomous Drivingcited by 12
  12. 2025Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generationcited by 10
38 more
  1. 2026MLLM-4D: Towards Visual-based Spatial-Temporal Intelligencecited by 10
  2. 2026MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentationcited by 8
  3. 2026Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Mapscited by 8
  4. 2025Part-X-MLLM: Part-aware 3D Multimodal Large Language Modelcited by 7
  5. 2026Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understandingcited by 7
  6. 2025Cook and Clean Together: Teaching Embodied Agents for Parallel Task Executioncited by 6
  7. 2025MoE3D: Mixture of Experts meets Multi-Modal 3D Understandingcited by 6
  8. 2025Toward Ambulatory Vision: Learning Visually-Grounded Active View Selectioncited by 6
  9. 2026Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligencecited by 6
  10. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 5
  11. 2025Vision-Language Memory for Spatial Reasoningcited by 5
  12. 2025SpatialMosaic: A Multiview VLM Dataset for Partial Visibilitycited by 5
  13. 2026Allocentric Perceiver: Disentangling Allocentric Reasoning from Egocentric Visual Priors via Frame Instantiationcited by 5
  14. 2025VaseVQA-3D: Benchmarking 3D VLMs on Ancient Greek Potterycited by 4
  15. 2025Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Modelscited by 3
  16. 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modelingcited by 3
  17. 2025EagleVision: A Dual-Stage Framework with BEV-grounding-based Chain-of-Thought for Spatial Intelligencecited by 3
  18. 2026Open-Vocabulary Functional 3D Human-Scene Interaction Generationcited by 3
  19. 20263DGSNav: Enhancing Vision-Language Model Reasoning for Object Navigation via Active 3D Gaussian Splattingcited by 3
  20. 2025Exploring 3D Reasoning-Driven Planning: From Implicit Human Intentions to Route-Aware Activity Planningcited by 2
  21. 2025GraspCoT: Integrating Physical Property Reasoning for 6-DoF Grasping under Flexible Language Instructionscited by 2
  22. 2025Point Linguist Model: Segment Any Object via Bridged Large 3D-Language Modelcited by 2
  23. 2026PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Modelscited by 2
  24. 2026Point Cloud as a Foreign Language for Multi-modal Large Language Modelcited by 2
  25. 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Modelscited by 2
  26. 2026Distilling Neuro-Symbolic Programs into 3D Multi-modal LLMscited by 2
  27. 2025Imagine in Space: Exploring the Frontier of Spatial Intelligence and Reasoning Efficiency in Vision Language Modelscited by 1
  28. 2025Error-Driven Scene Editing for 3D Grounding in Large Language Modelscited by 1
  29. 2025SpatialReasoner: Active Perception for Large-Scale 3D Scene Understandingcited by 1
  30. 2026Direction-aware 3D Large Multimodal Modelscited by 1
  31. 2026Efficient Encoder-Free Fourier-based 3D Large Multimodal Modelcited by 1
  32. 20263D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understandingcited by 1
  33. 20263D-IDE: 3D Implicit Depth Emergentcited by 1
  34. 20263D Segmentation Using Viewpoint-Dependent Spatial Relationshipscited by 1
  35. 2026Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Modelscited by 1
  36. 2026PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understandingcited by 1
  37. 2026Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priorscited by 1
  38. 2025Disc3D: Automatic Curation of High-Quality 3D Dialog Data via Discriminative Object Referringcited by 0

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness
  2. 2025LLaVA3: Representing 3D Scenes like a Cubist Painter to Boost 3D Scene Understanding of VLMs
  3. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  4. 20233D-LLM: Injecting the 3D World into Large Language Models
  5. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes
  6. 2026Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
  7. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
  8. 2025Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction-Reasoning Synergy