mlsanityData to 14 Oct 2026

PLA: Language-Driven Open-Vocabulary 3D Scene Understanding

arXiv 2211.16312 ↗First posted 29 Nov 2022cs.CVcited by 177

Not on the map. Why?

Cited by 26 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Open-vocabulary scene understanding aims to localize and recognize unseen categories beyond the annotated label space. The recent breakthrough of 2D open-vocabulary perception is largely driven by Internet-scale paired image-text data with rich vocabulary concepts. However, this success cannot be directly transferred to 3D scenarios due to the inaccessibility of large-scale 3D-text pairs. To this end, we propose to distill knowledge encoded in pre-trained vision-language (VL) foundation models through captioning multi-view images from 3D, which allows explicitly associating 3D and semantic-rich captions. Further, to foster coarse-to-fine visual-semantic representation learning from captions, we design hierarchical 3D-caption pairs, leveraging geometric constraints between 3D scenes and multi-view images. Finally, by employing contrastive learning, the model learns language-aware embeddings that connect 3D and text for open-vocabulary tasks. Our method not only remarkably outperforms baseline methods by 25.8% ~ 44.7% hIoU and 14.5% ~ 50.4% hAP50 in open-vocabulary semantic and instance segmentation, but also shows robust transferability on challenging zero-shot domain transfer tasks. See the project website at https://dingry.github.io/projects/PLA.

Key facts. Cited by 177 papers in the map (data through 14 Oct 2026). Builds on 46 of its 56 references that are in the map. First posted on arXiv 29 Nov 2022.

Builds on

46 of 56 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2017On Calibration of Modern Neural Networkscited by 4,960
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  9. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  10. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  11. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  12. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,205
34 more
  1. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  2. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  3. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 748
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  5. 2018Open3D: A Modern Library for 3D Data Processingcited by 721
  6. 2022Language-driven Semantic Segmentationcited by 717
  7. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  8. 2021Extract Free Dense Labels from CLIPcited by 583
  9. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  10. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  11. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  12. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  13. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 321
  14. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  15. 2019Zero-Shot Semantic Segmentationcited by 245
  16. 2017Submanifold Sparse Convolutional Networkscited by 223
  17. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  18. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 214
  19. 2021PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Cloudscited by 211
  20. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 200
  21. 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 167
  22. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 160
  23. 2019Learning Object Bounding Boxes for 3D Instance Segmentation on Point Cloudscited by 156
  24. 2018GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloudcited by 146
  25. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 146
  26. 2022PromptDet: Towards Open-vocabulary Detection using Uncurated Imagescited by 133
  27. 2016BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integrationcited by 116
  28. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  29. 2018Recurrent Slice Networks for 3D Segmentation of Point Cloudscited by 82
  30. 2019Neural Point Cloud Rendering via Multi-Plane Projectioncited by 60
  31. 2021Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Cloudscited by 55
  32. 2021Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentationcited by 53
  33. 2019Transductive Zero-Shot Learning for 3D Point Cloud Classificationcited by 39
  34. 2022Answer-Me: Multi-Task Open-Vocabulary Visual Question Answeringcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 41; 2024: 61; 2025: 52; 2026: 23.

177 papers.

Sort:

By area:

  1. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  2. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  3. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  4. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  5. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 210
  6. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 208
  7. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 184
  8. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  9. 2023Segment Any 3D Gaussianscited by 161
  10. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 147
  11. 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 145
  12. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
38 more
  1. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
  2. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  3. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  4. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 124
  5. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  6. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  7. 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 94
  8. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  9. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  10. 2024Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splattingcited by 82
  11. 2023OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Datacited by 81
  12. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 75
  13. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  14. 2023Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labelscited by 64
  15. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  16. 2023Segment Anything in 3D with Radiance Fieldscited by 62
  17. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 61
  18. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  19. 2023Towards Label-free Scene Understanding by Vision Foundation Modelscited by 54
  20. 2023CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detectioncited by 52
  21. 2025SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretrainingcited by 52
  22. 2025Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registrationcited by 51
  23. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 49
  24. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  25. 2023SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentationcited by 44
  26. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 43
  27. 2023Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understandingcited by 38
  28. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  29. 2023SAM-guided Graph Cut for 3D Instance Segmentationcited by 35
  30. 2024V-IRL: Grounding Virtual Intelligence in Real Lifecited by 35
  31. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 35
  32. 2024N2F2: Hierarchical Scene Understanding with Nested Neural Feature Fieldscited by 34
  33. 2023Zero-Shot 3D Shape Correspondencecited by 33
  34. 2024Situational Awareness Matters in 3D Vision Language Reasoningcited by 33
  35. 2023UnScene3D: Unsupervised 3D Instance Segmentation for Indoor Scenescited by 31
  36. 2024ODIN: A Single Model for 2D and 3D Segmentationcited by 25
  37. 2023Interactive Segment Anything NeRF with Feature Imitationcited by 24
  38. 2023Object2Scene: Putting Objects in Context for Open-Vocabulary 3D Detectioncited by 24

50 most cited of 177 All 177 (JSON)