mlsanityData to 7 Oct 2026

Language-Grounded Indoor 3D Semantic Segmentation in the Wild

arXiv 2204.07761 ↗First posted 16 Apr 2022cs.CVcited by 243

Not on the map. Why?

Cited by 24 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in 3D semantic segmentation with deep neural networks have shown remarkable success, with rapid performance increase on available datasets. However, current 3D semantic segmentation benchmarks contain only a small number of categories -- less than 30 for ScanNet and SemanticKITTI, for instance, which are not enough to reflect the diversity of real environments (e.g., semantic image understanding covers hundreds to thousands of classes). Thus, we propose to study a larger vocabulary for 3D semantic segmentation with a new extended benchmark on ScanNet data with 200 class categories, an order of magnitude more than previously studied. This large number of class categories also induces a large natural class imbalance, both of which are challenging for existing 3D semantic segmentation methods. To learn more robust 3D features in this context, we propose a language-driven pre-training method to encourage learned 3D features that might have limited training examples to lie close to their pre-trained text embeddings. Extensive experiments show that our approach consistently outperforms state-of-the-art 3D pre-training for 3D semantic segmentation on our proposed benchmark (+9% relative mIoU), including limited-data scenarios with +25% relative mIoU using only 5% annotations.

Key facts. Cited by 243 papers in the map (data through 7 Oct 2026). Builds on 45 of its 57 references that are in the map. First posted on arXiv 16 Apr 2022.

Builds on

45 of 57 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  8. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  11. 2020Supervised Contrastive Learningcited by 2,803
  12. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
33 more
  1. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  2. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  3. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  4. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 806
  5. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  6. 2022Language-driven Semantic Segmentationcited by 711
  7. 2019RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Cloudscited by 584
  8. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 510
  9. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  10. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  11. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 340
  12. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  13. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  14. 2018SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filterscited by 261
  15. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  16. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  17. 2020Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contextscited by 210
  18. 2021Spatio-temporal Self-Supervised Representation Learning for 3D Point Cloudscited by 180
  19. 2020OccuSeg: Occupancy-aware 3D Instance Segmentationcited by 177
  20. 2021RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentationcited by 173
  21. 2020Overcoming Classifier Imbalance for Long-tail Object Detection with Balanced Group Softmaxcited by 152
  22. 2021Mix3D: Out-of-Context Data Augmentation for 3D Scenescited by 126
  23. 2018ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shapecited by 123
  24. 20183DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentationcited by 118
  25. 2021One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentationcited by 89
  26. 2021Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Cloudscited by 55
  27. 20214DContrast: Contrastive Learning with Dynamic Correspondences for 3D Scene Understandingcited by 54
  28. 2021Zero-Shot Learning on 3D Point Cloud Objects and Beyondcited by 38
  29. 2021RandomRooms: Unsupervised Pre-training from Synthetic Shapes and Randomized Layouts for 3D Object Detectioncited by 37
  30. 2016Survey of resampling techniques for improving classification performance in unbalanced datasetscited by 32
  31. 2021DropLoss for Long-Tail Instance Segmentationcited by 32
  32. 2019LU-Net: An Efficient Network for 3D LiDAR Point Cloud Semantic Segmentation Based on End-to-End-Learned 3D Features and U-Netcited by 5
  33. 2019Exploiting Synthetically Generated Data with Semi-Supervised Learning for Small and Imbalanced Datasetscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 56; 2024: 65; 2025: 72; 2026: 39.

243 papers.

Sort:

By area:

  1. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  2. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  3. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  4. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  5. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  6. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  7. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  8. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  9. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  10. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  11. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  12. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
38 more
  1. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  2. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  3. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  4. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
  5. 2022PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Modelscited by 117
  6. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  7. 2025Sonata: Self-Supervised Learning of Reliable Point Representationscited by 108
  8. 2023OctFormer: Octree-based Transformers for 3D Point Cloudscited by 105
  9. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 98
  10. 2023OneFormer3D: One Transformer for Unified Point Cloud Segmentationcited by 95
  11. 2023RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understandingcited by 93
  12. 2023SAI3D: Segment Any Instance in 3D Scenescited by 90
  13. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  14. 2023Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Trainingcited by 85
  15. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  16. 2023Open-Vocabulary Point-Cloud Object Detection without 3D Annotationcited by 81
  17. 2023OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Datacited by 81
  18. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  19. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 68
  20. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  21. 2023Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labelscited by 64
  22. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  23. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  24. 2023PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigmcited by 56
  25. 2024MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentationcited by 56
  26. 2023Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learningcited by 55
  27. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  28. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
  29. 2023CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detectioncited by 51
  30. 2024VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videoscited by 51
  31. 2025SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretrainingcited by 51
  32. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Modelscited by 50
  33. 2025Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registrationcited by 50
  34. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  35. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  36. 2023FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Modelscited by 47
  37. 2023Understanding Imbalanced Semantic Segmentation Through Neural Collapsecited by 46
  38. 2023Mask-Attention-Free Transformer for 3D Instance Segmentationcited by 45

50 most cited of 243 All 243 (JSON)