mlsanityData to 14 Oct 2026

Model2Scene: Learning 3D Scene Representation via Contrastive Language-CAD Models Pre-training

arXiv 2309.16956 ↗First posted 29 Sep 2023cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Current successful methods of 3D scene perception rely on the large-scale annotated point cloud, which is tedious and expensive to acquire. In this paper, we propose Model2Scene, a novel paradigm that learns free 3D scene representation from Computer-Aided Design (CAD) models and languages. The main challenges are the domain gaps between the CAD models and the real scene's objects, including model-to-scene (from a single model to the scene) and synthetic-to-real (from synthetic model to real scene's object). To handle the above challenges, Model2Scene first simulates a crowded scene by mixing data-augmented CAD models. Next, we propose a novel feature regularization operation, termed Deep Convex-hull Regularization (DCR), to project point features into a unified convex hull space, reducing the domain gap. Ultimately, we impose contrastive loss on language embedding and the point features of CAD models to pre-train the 3D network. Extensive experiments verify the learned 3D scene representation is beneficial for various downstream tasks, including label-free 3D object salient detection, label-efficient 3D scene perception and zero-shot 3D semantic segmentation. Notably, Model2Scene yields impressive label-free 3D object salient detection with an average mAP of 46.08% and 55.49% on the ScanNet and S3DIS datasets, respectively. The code will be publicly available.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 72 of its 93 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

72 of 93 references.

References · 72

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  6. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  7. 2017Adversarial Discriminative Domain Adaptationcited by 1,311
  8. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  9. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  10. 2019Large-Scale Long-Tailed Recognition in an Open Worldcited by 705
  11. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  12. 2014Neural Turing Machinescited by 644
38 more
  1. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  2. 2019Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detectioncited by 564
  3. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  4. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 504
  5. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  6. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 423
  7. 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 406
  8. 2015Attention-Based Models for Speech Recognitioncited by 401
  9. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  10. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  11. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  12. 2020Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contextscited by 210
  13. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 181
  14. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  15. 2018GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloudcited by 146
  16. 2019xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentationcited by 129
  17. 2021LiDAR R-CNN: An Efficient and Universal 3D Object Detectorcited by 129
  18. 2023Robo3D: Towards Robust and Reliable 3D Perception against Corruptionscited by 129
  19. 2023Rethinking Range View Representation for LiDAR Segmentationcited by 128
  20. 2021Hierarchical Aggregation for 3D Instance Segmentationcited by 127
  21. 2021Mix3D: Out-of-Context Data Augmentation for 3D Scenescited by 126
  22. 2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
  23. 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 125
  24. 2020H3DNet: 3D Object Detection Using Hybrid Geometric Primitivescited by 123
  25. 2020Complete & Label: A Domain Adaptation Approach to Semantic Segmentation of LiDAR Point Cloudscited by 116
  26. 2019PointDAN: A Multi-Scale 3D Domain Adaption Network for Point Cloud Representationcited by 114
  27. 2020PointMixup: Augmentation for Point Cloudscited by 104
  28. 20203DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detectioncited by 89
  29. 2021Guided Point Contrastive Learning for Semi-supervised Point Cloud Semantic Segmentationcited by 87
  30. 2020LiDAR-based Panoptic Segmentation via Dynamic Shifting Networkcited by 84
  31. 2020Few-shot 3D Point Cloud Semantic Segmentationcited by 77
  32. 2020SSN: Shape Signature Networks for Multi-class Object Detection from Point Cloudscited by 74
  33. 2021PointCutMix: Regularization Strategy for Point Cloud Classificationcited by 69
  34. 2020PointAugment: an Auto-Augmentation Framework for Point Cloud Classificationcited by 65
  35. 2021Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR-based Perceptioncited by 57
  36. 2023UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebasecited by 57
  37. 2021Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Cloudscited by 55
  38. 2021SSPC-Net: Semi-supervised Semantic 3D Point Cloud Segmentation Networkcited by 54

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2023Winning Prize Comes from Losing Tickets: Improve Invariant Learning by Exploring Variant Parameters for Out-of-Distribution Generalizationcited by 12
  2. 2024PanoSLAM: Panoptic 3D Scene Reconstruction via Gaussian SLAMcited by 3
  3. 2024HUNTER: Unsupervised Human-centric 3D Detection via Transferring Knowledge from Synthetic Instances to Real Scenescited by 2