mlsanityData to 14 Oct 2026

Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes

arXiv 2308.08769 ↗First posted 17 Aug 2023cs.CVcited by 129

Not on the map. Why?

Cited by 57 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

3D scene understanding has gained significant attention due to its wide range of applications. However, existing methods for 3D scene understanding are limited to specific downstream tasks, which hinders their practicality in real-world applications. This paper presents Chat-3D, which combines the 3D visual perceptual ability of pre-trained 3D representations and the impressive reasoning and conversation capabilities of advanced LLMs to achieve the first universal dialogue systems for 3D scenes. Specifically, we align 3D representations into the feature space of LLMs, thus enabling LLMs to perceive the 3D world. Given the scarcity of 3D scene-text data, we propose a three-stage training strategy to efficiently utilize the available data for better alignment. To enhance the reasoning ability and develop a user-friendly interaction scheme, we further construct a high-quality object-centric 3D instruction dataset and design an associated object-centric prompt. Our experiments show that Chat-3D achieves an impressive ability to comprehend diverse instructions for 3D scenes, engage in intricate spatial reasoning, and incorporate external knowledge into its responses. Chat-3D achieves a 75.6% relative score compared with GPT-4 on the constructed instruction dataset.

Key facts. Cited by 129 papers in the map (data through 14 Oct 2026). Builds on 52 of its 59 references that are in the map. First posted on arXiv 17 Aug 2023.

Builds on

52 of 59 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  12. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2015VQA: Visual Question Answeringcited by 2,506
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  5. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  7. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  9. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  10. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  13. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  14. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  15. 2017Dense-Captioning Events in Videoscited by 793
  16. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  17. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  18. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  19. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  20. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  21. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  22. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  23. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 321
  24. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  25. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 271
  26. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 267
  27. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 264
  28. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  29. 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
  30. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 184
  31. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 162
  32. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 144
  33. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  34. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 122
  35. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  36. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  37. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  38. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 31; 2025: 50; 2026: 39.

129 papers.

Sort:

By area:

  1. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  2. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  3. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  4. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  5. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  6. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 138
  7. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  8. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  9. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 103
  10. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  11. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 70
  12. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 63
38 more
  1. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 51
  2. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLMcited by 42
  3. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  4. 2023LLMR: Real-time Prompting of Interactive Worlds using Large Language Modelscited by 39
  5. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 39
  6. 2024LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferencescited by 38
  7. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 38
  8. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  9. 20253D Aware Region Prompted Vision Language Modelcited by 33
  10. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 33
  11. 2023M3DBench: Let's Instruct Large Models with Multi-modal 3D Promptscited by 32
  12. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understandingcited by 32
  13. 20253D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worldscited by 30
  14. 2024Agent3D-Zero: An Agent for Zero-shot 3D Understandingcited by 29
  15. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 27
  16. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  17. 2025SplatTalk: 3D VQA with Gaussian Splattingcited by 24
  18. 2024Affordance-Guided Reinforcement Learning via Visual Promptingcited by 23
  19. 2024GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMscited by 23
  20. 2023Extending Multi-modal Contrastive Representationscited by 21
  21. 2024A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulationcited by 21
  22. 2024FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusioncited by 19
  23. 2024Chat-Edit-3D: Interactive 3D Scene Editing via Text Promptscited by 19
  24. 2024SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Mannerscited by 18
  25. 2025LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understandingcited by 17
  26. 2024Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intentioncited by 16
  27. 2025Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMscited by 16
  28. 2025Exploring the Potential of Encoder-free Architectures in 3D LMMscited by 16
  29. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentationcited by 16
  30. 20233DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4Vcited by 15
  31. 2024The Effects of Embodiment and Personality Expression on Learning in LLM-based Educational Agentscited by 13
  32. 2023Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluationcited by 12
  33. 2024LLM-enhanced Scene Graph Learning for Household Rearrangementcited by 12
  34. 2025GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyondcited by 12
  35. 2024Grounding 3D Scene Affordance From Egocentric Interactionscited by 11
  36. 2024RG-SAN: Rule-Guided Spatial Awareness Network for End-to-End 3D Referring Expression Segmentationcited by 11
  37. 2025SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenescited by 11
  38. 2026CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelingcited by 11

50 most cited of 129 All 129 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  2. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning
  3. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
  4. 2025Masking Matters: Unlocking the Spatial Reasoning Capabilities of LLMs for 3D Scene-Language Understanding
  5. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning
  6. 2024SceneGPT: A Language Model for 3D Scene Understanding
  7. 20233D-LLM: Injecting the 3D World into Large Language Models
  8. 2025Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models