mlsanityData to 7 Oct 2026

RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding

arXiv 2304.00962 ↗First posted 3 Apr 2023cs.CV cs.AIcited by 93

Not on the map. Why?

Abstract (arXiv)

We propose a lightweight and scalable Regional Point-Language Contrastive learning framework, namely RegionPLC, for open-world 3D scene understanding, aiming to identify and recognize open-set objects and categories. Specifically, based on our empirical studies, we introduce a 3D-aware SFusion strategy that fuses 3D vision-language pairs derived from multiple 2D foundation models, yielding high-quality, dense region-level language descriptions without human 3D annotations. Subsequently, we devise a region-aware point-discriminative contrastive learning objective to enable robust and effective 3D learning from dense regional language supervision. We carry out extensive experiments on ScanNet, ScanNet200, and nuScenes datasets, and our model outperforms prior 3D open-world scene understanding approaches by an average of 17.2% and 9.1% for semantic and instance segmentation, respectively, while maintaining greater scalability and lower resource demands. Furthermore, our method has the flexibility to be effortlessly integrated with language models to enable open-ended grounded 3D reasoning without extra task-specific training. Code is available at https://github.com/CVMI-Lab/PLA.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Builds on 50 of its 67 references that are in the map. First posted on arXiv 3 Apr 2023.

Builds on

50 of 67 references.

References · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  9. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  10. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  11. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  12. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
38 more
  1. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  2. 2021Grounded Language-Image Pre-trainingcited by 1,147
  3. 2016End to End Learning for Self-Driving Carscited by 1,000
  4. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  5. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  6. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  7. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  8. 2022Language-driven Semantic Segmentationcited by 711
  9. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
  10. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  11. 2021Extract Free Dense Labels from CLIPcited by 580
  12. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  13. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  14. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  15. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  16. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  17. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  18. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  19. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  20. 2021PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Cloudscited by 211
  21. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 197
  22. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  23. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  24. 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 166
  25. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  26. 2019Learning Object Bounding Boxes for 3D Instance Segmentation on Point Cloudscited by 156
  27. 2018GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloudcited by 145
  28. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
  29. 2022Focal Sparse Convolutional Networks for 3D Object Detectioncited by 142
  30. 2022PromptDet: Towards Open-vocabulary Detection using Uncurated Imagescited by 133
  31. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 121
  32. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  33. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  34. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69
  35. 2022Don't Stop Learning: Towards Continual Learning for the CLIP Modelcited by 61
  36. 2021Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Cloudscited by 55
  37. 2019Transductive Zero-Shot Learning for 3D Point Cloud Classificationcited by 39
  38. 2023Top-Down Beats Bottom-Up in 3D Instance Segmentationcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 24; 2025: 38; 2026: 18.

93 papers.

Sort:

By area:

  1. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  2. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  3. 2023Segment Any 3D Gaussianscited by 158
  4. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  5. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  6. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  7. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  8. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  9. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  10. 2023Segment Anything in 3D with Radiance Fieldscited by 61
  11. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  12. 2025SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretrainingcited by 51
38 more
  1. 2025Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registrationcited by 50
  2. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Modelscited by 41
  3. 2023Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understandingcited by 38
  4. 2024V-IRL: Grounding Virtual Intelligence in Real Lifecited by 35
  5. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 34
  6. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  7. 2024EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensingcited by 29
  8. 2024TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understandingcited by 26
  9. 2025Embodied AI: From LLMs to World Modelscited by 23
  10. 2023A Survey of Label-Efficient Deep Learning for 3D Point Cloudscited by 20
  11. 2025Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentationcited by 20
  12. 2023Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-trainingcited by 17
  13. 2024Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillationcited by 17
  14. 2025Generalized Few-shot 3D Point Cloud Segmentation with Vision-Language Modelcited by 17
  15. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentationcited by 16
  16. 2024UniM-OV3D: Uni-Modality Open-Vocabulary 3D Scene Understanding with Fine-Grained Feature Representationcited by 15
  17. 2024Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understandingcited by 15
  18. 2025PartSAM: A Scalable Promptable Part Segmentation Model Trained on Native 3D Datacited by 15
  19. 2025SceneSplat++: A Large Dataset and Comprehensive Benchmark for Language Gaussian Splattingcited by 14
  20. 2026Multiplayer Interactive World Models with Representation Autoencoderscited by 14
  21. 20243D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillationcited by 13
  22. 2024Segment Any 3D Object with Languagecited by 13
  23. 2025Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understandingcited by 13
  24. 2023Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluationcited by 12
  25. 2024Can 3D Vision-Language Models Truly Understand Natural Language?cited by 12
  26. 2025Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understandingcited by 10
  27. 2024Multimodal 3D Reasoning Segmentation with Complex Scenescited by 8
  28. 2025GaussianGraph: 3D Gaussian-based Scene Graph Generation for Open-world Scene Understandingcited by 8
  29. 2025LogoSP: Local-global Grouping of Superpoints for Unsupervised Semantic Segmentation of 3D Point Cloudscited by 8
  30. 2025Details Matter for Indoor Open-vocabulary 3D Instance Segmentationcited by 8
  31. 2025COS3D: Collaborative Open-Vocabulary 3D Segmentationcited by 8
  32. 2025Chorus: Multi-Teacher Pretraining for Holistic 3D Gaussian Scene Encodingcited by 8
  33. 2025VLScene: Vision-Language Guidance Distillation for Camera-Based 3D Semantic Scene Completioncited by 6
  34. 2025HIS-GPT: Towards 3D Human-In-Scene Multimodal Understandingcited by 6
  35. 2025Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scalecited by 6
  36. 2024OpenScan: A Benchmark for Generalized Open-Vocabulary 3D Scene Understandingcited by 5
  37. 2024XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentationcited by 5
  38. 2025SAS: Segment Any 3D Scene with Integrated 2D Priorscited by 5

50 most cited of 93 All 93 (JSON)