mlsanityData to 14 Oct 2026

Multi3DRefer: Grounding Text Description to Multiple 3D Objects

arXiv 2309.05251 ↗First posted 11 Sep 2023cs.CVcited by 146

Not on the map. Why?

Cited by 49 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce the task of localizing a flexible number of objects in real-world 3D scenes using natural language descriptions. Existing 3D visual grounding tasks focus on localizing a unique object given a text description. However, such a strict setting is unnatural as localizing potentially multiple objects is a common need in real-world scenarios and robotic tasks (e.g., visual navigation and object rearrangement). To address this setting we propose Multi3DRefer, generalizing the ScanRefer dataset and task. Our dataset contains 61926 descriptions of 11609 objects, where zero, single or multiple target objects are referenced by each description. We also introduce a new evaluation metric and benchmark methods from prior work to enable further investigation of multi-modal 3D scene understanding. Furthermore, we develop a better baseline leveraging 2D features from CLIP by rendering object proposals online with contrastive learning, which outperforms the state of the art on the ScanRefer benchmark.

Key facts. Cited by 146 papers in the map (data through 14 Oct 2026). Builds on 50 of its 68 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

50 of 68 references.

References · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  9. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  10. 2021Grounded Language-Image Pre-trainingcited by 1,156
  11. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  12. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
38 more
  1. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 784
  2. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  3. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  4. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  5. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 321
  6. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  7. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 271
  8. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  9. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 195
  10. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  11. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 165
  12. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  13. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 141
  14. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  15. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 139
  16. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  17. 2015Natural Language Object Retrievalcited by 124
  18. 2021SAT: 2D Semantics Assisted Training for 3D Visual Groundingcited by 124
  19. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 122
  20. 2020PhraseCut: Language-based Image Segmentation in the Wildcited by 116
  21. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 110
  22. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 99
  23. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 89
  24. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  25. 2017Grounding Referring Expressions in Images by Variational Contextcited by 82
  26. 2021Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloudcited by 81
  27. 2022UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Groundingcited by 69
  28. 2023NS3D: Neuro-Symbolic Grounding of 3D Objects and Relationscited by 57
  29. 2019ShapeGlot: Learning Language for Shape Differentiationcited by 54
  30. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  31. 2021Language Grounding with 3D Objectscited by 43
  32. 2022Look Around and Refer: 2D Synthetic Semantics Knowledge Distillation for 3D Visual Groundingcited by 41
  33. 2016Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cuescited by 33
  34. 2022ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenescited by 32
  35. 2021PartGlot: Learning Shape Part Segmentation from Language Reference Gamescited by 30
  36. 2022Open-Vocabulary 3D Detection via Image-level Class and Debiased Cross-modal Contrastive Learningcited by 23
  37. 2022FindIt: Generalized Localization with Natural Language Queriescited by 13
  38. 2022A Unified Mutual Supervision Framework for Referring Expression Segmentation and Generationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 32; 2025: 65; 2026: 44.

146 papers.

Sort:

By area:

  1. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 183
  2. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  3. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  4. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 138
  5. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  6. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  7. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  8. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 80
  9. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  10. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 70
  11. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 68
  12. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 63
38 more
  1. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 54
  2. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 51
  3. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 51
  4. 2024Generating Human Motion in 3D Scenes from Text Descriptionscited by 49
  5. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 49
  6. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  7. 2024TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenescited by 40
  8. 2024MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotationscited by 40
  9. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 35
  10. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  11. 2024MiKASA: Multi-Key-Anchor & Scene-Aware Transformer for 3D Visual Groundingcited by 29
  12. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 27
  13. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  14. 2024A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directionscited by 25
  15. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Modelscited by 25
  16. 2025OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understandingcited by 24
  17. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 23
  18. 2025Multimodal Referring Segmentation: A Surveycited by 22
  19. 2025ReferSplat: Referring Segmentation in 3D Gaussian Splattingcited by 20
  20. 2025RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenescited by 19
  21. 2025Reasoning in Space via Grounding in the Worldcited by 19
  22. 20243D-GRES: Generalized 3D Referring Expression Segmentationcited by 18
  23. 2025LLaVA-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understandingcited by 17
  24. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentationcited by 16
  25. 2025Unifying 2D and 3D Vision-Language Understandingcited by 15
  26. 2023Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluationcited by 12
  27. 2024Can 3D Vision-Language Models Truly Understand Natural Language?cited by 12
  28. 2025SPAZER: Spatial-Semantic Progressive Reasoning Agent for Zero-shot 3D Visual Groundingcited by 12
  29. 20253EED: Ground Everything Everywhere in 3Dcited by 12
  30. 2025AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referringcited by 10
  31. 2025Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understandingcited by 10
  32. 2025Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generationcited by 10
  33. 2023Uni3DL: Unified Model for 3D and Language Understandingcited by 9
  34. 2024Task-oriented Sequential Grounding and Navigation in 3D Scenescited by 9
  35. 2024Multimodal 3D Reasoning Segmentation with Complex Scenescited by 8
  36. 2025Ges3ViG: Incorporating Pointing Gestures into Language-Based 3D Visual Grounding for Embodied Reference Understandingcited by 8
  37. 2025Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMscited by 8
  38. 2025ViewSRD: 3D Visual Grounding via Structured Multi-View Decompositioncited by 8

50 most cited of 146 All 146 (JSON)