mlsanityData to 7 Oct 2026

Grounded 3D-LLM with Referent Tokens

arXiv 2405.10370 ↗First posted 16 May 2024cs.CVcited by 111

Not on the map. Why?

Cited by 57 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Prior studies on 3D scene understanding have primarily developed specialized models for specific tasks or required task-specific fine-tuning. In this study, we propose Grounded 3D-LLM, which explores the potential of 3D large multi-modal models (3D LMMs) to consolidate various 3D vision tasks within a unified generative framework. The model uses scene referent tokens as special noun phrases to reference 3D scenes, enabling it to handle sequences that interleave 3D and textual data. Per-task instruction-following templates are employed to ensure natural and diversity in translating 3D vision tasks into language formats. To facilitate the use of referent tokens in subsequent language modeling, we provide a large-scale, automatically curated grounded scene-text dataset with over 1 million phrase-to-region correspondences and introduce Contrastive Language-Scene Pre-training (CLASP) to perform phrase-level scene-text alignment using this data. Our comprehensive evaluation covers open-ended tasks like dense captioning and 3D question answering, alongside close-ended tasks such as object detection and language grounding. Experiments across multiple 3D benchmarks reveal the leading performance and the broad applicability of Grounded 3D-LLM. Code and datasets are available at the https://groundedscenellm.github.io/grounded_3d-llm.github.io.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Builds on 86 of its 100 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

86 of 100 references.

References · 86

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  12. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
38 more
  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  5. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  6. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  7. 2021Grounded Language-Image Pre-trainingcited by 1,147
  8. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  9. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  10. 2022A Generalist Agentcited by 802
  11. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  12. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  13. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  14. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  15. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  16. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  17. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  18. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  19. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  20. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  21. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  22. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  23. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  24. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  25. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  26. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  27. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  28. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  29. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  30. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  31. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  32. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  33. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 231
  34. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  35. 2021Group-Free 3D Object Detection via Transformerscited by 215
  36. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  37. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  38. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 197

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 61; 2026: 39.

111 papers.

Sort:

By area:

  1. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  2. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  3. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  4. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 127
  5. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  6. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  7. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  8. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  9. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  10. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  11. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLMcited by 41
  12. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
38 more
  1. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 34
  2. 2025AutoPartGen: Autogressive 3D Part Generation and Discoverycited by 33
  3. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  4. 20253D Aware Region Prompted Vision Language Modelcited by 31
  5. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 31
  6. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  7. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  8. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 26
  9. 20253DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Modelcited by 25
  10. 2025SplatTalk: 3D VQA with Gaussian Splattingcited by 24
  11. 2025URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Modelcited by 20
  12. 2025MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Meshcited by 18
  13. 2025Reasoning in Space via Grounding in the Worldcited by 18
  14. 2024Functionality understanding and segmentation in 3D scenescited by 17
  15. 2025LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understandingcited by 17
  16. 2025Exploring the Potential of Encoder-free Architectures in 3D LMMscited by 16
  17. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentationcited by 16
  18. 2025Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview imagescited by 15
  19. 2025AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Modelscited by 15
  20. 2025GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyondcited by 12
  21. 2025SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenescited by 11
  22. 2025SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Modelscited by 10
  23. 20253D Question Answering via only 2D Vision-Language Modelscited by 10
  24. 2025UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encodingcited by 9
  25. 2024Multimodal 3D Reasoning Segmentation with Complex Scenescited by 8
  26. 2024PerLA: Perceptive 3D Language Assistantcited by 8
  27. 2025Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMscited by 8
  28. 2026MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentationcited by 8
  29. 2026CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelingcited by 8
  30. 2025Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Modelscited by 7
  31. 2025SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenescited by 7
  32. 2025HIS-GPT: Towards 3D Human-In-Scene Multimodal Understandingcited by 6
  33. 2025Cook and Clean Together: Teaching Embodied Agents for Parallel Task Executioncited by 6
  34. 2025MoE3D: Mixture of Experts meets Multi-Modal 3D Understandingcited by 6
  35. 2026Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understandingcited by 6
  36. 2026Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligencecited by 6
  37. 20243D-AVS: LiDAR-based 3D Auto-Vocabulary Segmentationcited by 5
  38. 2025Empowering Large Language Models with 3D Situation Awarenesscited by 5

50 most cited of 111 All 111 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  2. 20233D-LLM: Injecting the 3D World into Large Language Models
  3. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding
  4. 2026Lens3D: Target-Conditioned Visual Foveation for Fine-Grained 3D Understanding
  5. 2023Weakly-Supervised 3D Visual Grounding based on Visual Language Alignment
  6. 2025Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning without 3D Annotations
  7. 2024Transcrib3D: 3D Referring Expression Resolution through Large Language Models
  8. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding