Disc3D: Automatic Curation of High-Quality 3D Dialog Data via Discriminative Object Referring
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
3D Multi-modal Large Language Models (MLLMs) still lag behind their 2D peers, largely because large-scale, high-quality 3D scene-dialogue datasets remain scarce. Prior efforts hinge on expensive human annotation and leave two key ambiguities unresolved: viewpoint ambiguity, where spatial language presumes unknown camera poses, and object referring ambiguity, where non-exclusive descriptions blur the line between targets and distractors. We therefore present a fully automated pipeline that converts raw 3D scans into unambiguous, high-quality dialogue data at a fraction of the previous cost. By synergizing rule-based constraints with 2D MLLMs and LLMs, the pipeline enables controllable, scalable generation without human intervention. The pipeline comprises four stages: (1) meta-annotation collection harvesting object-, frame-, and scene-level captions, (2) scene graph construction with relation correction to capture proximal object relations, (3) discriminative object referring that generates exclusive and compact descriptions, and (4) multi-task data generation synthesizing diverse dialogues. Our pipeline systematically mitigates inherent flaws in source datasets and produces the final Disc3D dataset, over 2 million samples in 25K hybrid 3D scenes, spanning scene, view, and object captioning, visual grounding, and five object-centric QA tasks. Extensive experiments demonstrate that training with Disc3D yields consistent, significant improvements on both public benchmarks and our multifaceted Disc3D-QA tasks. Code, data, and models will be publicly available.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 47 references that are in the map. First posted on arXiv 24 Nov 2025.
Builds on
40 of 47 references.
Closest ancestors · 9
Same direction, newest first
- 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness2 Apr
- 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning1 Mar
- 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models2 Jan
- 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer2 Jan
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
- 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
- 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness26 Sep
- 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generation5 Dec
- 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
Other references · 31
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
- 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
- 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
- 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
- 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
- 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
- 2025Kimi-VL Technical Reportcited by 388
- 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
- 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
19 more
- 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
- 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
- 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
- 2019Structured3D: A Large Photo-realistic Dataset for Structured 3D Modelingcited by 286
- 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
- 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 188
- 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
- 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
- 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 141
- 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
- 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
- 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 110
- 2024SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Modelcited by 96
- 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
- 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
- 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
- 2024Cubify Anything: Scaling Indoor 3D Object Detectioncited by 39
- 2024Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentationcited by 30
- 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 23
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes
- 2024Grounded 3D-LLM with Referent Tokens
- 2026PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understanding
- 2026Ground3D-LMM: Fine-Grained 3D Point Grounding and Spatial Reasoning with LMM
- 2025Text-Scene: A Scene-to-Language Parsing Framework for 3D Scene Understanding
- 2026Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM
- 2026MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue
- 2025DenseScan: Advancing 3D Scene Understanding with 2D Dense Annotation