mlsanityData to 14 Oct 2026

Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers

arXiv 2312.08168 ↗First posted 13 Dec 2023cs.CVcited by 175

Not on the map. Why?

Cited by 85 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advancements in 3D Large Language Models (LLMs) have demonstrated promising capabilities for 3D scene understanding. However, previous methods exhibit deficiencies in general referencing and grounding capabilities for intricate scene comprehension. In this paper, we introduce the use of object identifiers and object-centric representations to interact with scenes at the object level. Specifically, we decompose the input 3D scene into a set of object proposals, each assigned a unique identifier token, which enables efficient object referencing and grounding during user-assistant interactions. Given the scarcity of scene-language data, we model the scene embeddings as a sequence of explicit object-level embeddings, derived from semantic-rich 2D or 3D representations. By employing object identifiers, we transform diverse 3D scene-language tasks into a unified question-answering format, facilitating joint training without the need for additional task-specific heads. With minimal fine-tuning on all downstream tasks, our model significantly outperforms existing methods on benchmarks including ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.

Key facts. Cited by 175 papers in the map (data through 14 Oct 2026). Builds on 64 of its 84 references that are in the map. First posted on arXiv 13 Dec 2023.

Builds on

64 of 84 references.

References · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 857
38 more
  1. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  2. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  3. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  4. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  5. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  6. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  7. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  8. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  9. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  10. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 321
  11. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  12. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
  13. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 267
  14. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 264
  15. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 237
  16. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  17. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  18. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  19. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 200
  20. 2023Tracking Anything with Decoupled Video Segmentationcited by 197
  21. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  22. 2020Unsupervised Point Cloud Pre-Training via Occlusion Completioncited by 186
  23. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 184
  24. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  25. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  26. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 144
  27. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  28. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 141
  29. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  30. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  31. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 129
  32. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  33. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  34. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 122
  35. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 122
  36. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  37. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 110
  38. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 99

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 82; 2026: 75.

175 papers.

Sort:

By area:

  1. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  2. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 211
  3. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 183
  4. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 166
  5. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 138
  6. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 120
  7. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 118
  8. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  9. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  10. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 96
  11. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 68
  12. 2025SpatialLM: Training Large Language Models for Structured Indoor Modelingcited by 65
38 more
  1. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  2. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  3. 2024Multi-modal Situated Reasoning in 3D Scenescited by 51
  4. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 51
  5. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  6. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  7. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 40
  8. 2024LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferencescited by 38
  9. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 38
  10. 2025Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spacescited by 33
  11. 20253D Aware Region Prompted Vision Language Modelcited by 33
  12. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 33
  13. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 27
  14. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  15. 20253DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Modelcited by 25
  16. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understandingcited by 24
  17. 2025SplatTalk: 3D VQA with Gaussian Splattingcited by 24
  18. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 23
  19. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 23
  20. 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligencecited by 23
  21. 2025Reasoning in Space via Grounding in the Worldcited by 19
  22. 2026PointCoT: A Multi-modal Benchmark for Explicit 3D Geometric Reasoningcited by 19
  23. 2024Intent3D: 3D Object Detection in RGB-D Scans Based on Human Intentioncited by 16
  24. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentationcited by 16
  25. 2025NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Predictioncited by 16
  26. 20253D CoCa: Contrastive Learners are 3D Captionerscited by 13
  27. 2025DenseGrounding: Improving Dense Language-Vision Semantics for Ego-Centric 3D Visual Groundingcited by 12
  28. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Groundingcited by 12
  29. 2025GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyondcited by 12
  30. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answeringcited by 11
  31. 2025SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenescited by 11
  32. 2025Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understandingcited by 10
  33. 20253D Question Answering via only 2D Vision-Language Modelscited by 10
  34. 2025Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMscited by 10
  35. 2025UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encodingcited by 10
  36. 2025KeySG: Hierarchical Keyframe-Based 3D Scene Graphscited by 10
  37. 2026MLLM-4D: Towards Visual-based Spatial-Temporal Intelligencecited by 10
  38. 2024Task-oriented Sequential Grounding and Navigation in 3D Scenescited by 9

50 most cited of 175 All 175 (JSON)