mlsanityData to 14 Oct 2026

CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP

arXiv 2301.04926 ↗First posted 12 Jan 2023cs.CVcited by 181

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) achieves promising results in 2D zero-shot and few-shot learning. Despite the impressive performance in 2D, applying CLIP to help the learning in 3D scene understanding has yet to be explored. In this paper, we make the first attempt to investigate how CLIP knowledge benefits 3D scene understanding. We propose CLIP2Scene, a simple yet effective framework that transfers CLIP knowledge from 2D image-text pre-trained models to a 3D point cloud network. We show that the pre-trained 3D network yields impressive performance on various downstream tasks, i.e., annotation-free and fine-tuning with labelled data for semantic segmentation. Specifically, built upon CLIP, we design a Semantic-driven Cross-modal Contrastive Learning framework that pre-trains a 3D network via semantic and spatial-temporal consistency regularization. For the former, we first leverage CLIP's text semantics to select the positive and negative point samples and then employ the contrastive loss to train the 3D network. In terms of the latter, we force the consistency between the temporally coherent point cloud features and their corresponding image features. We conduct experiments on SemanticKITTI, nuScenes, and ScanNet. For the first time, our pre-trained network achieves annotation-free 3D semantic segmentation with 20.8% and 25.08% mIoU on nuScenes and ScanNet, respectively. When fine-tuned with 1% or 100% labelled data, our method significantly outperforms other self-supervised methods, with improvements of 8% and 1% mIoU, respectively. Furthermore, we demonstrate the generalizability for handling cross-domain datasets. Code is publicly available https://github.com/runnanchen/CLIP2Scene.

Key facts. Cited by 181 papers in the map (data through 14 Oct 2026). Builds on 52 of its 67 references that are in the map. First posted on arXiv 12 Jan 2023.

Builds on

52 of 67 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  10. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
38 more
  1. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  2. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  3. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  4. 2021Extract Free Dense Labels from CLIPcited by 583
  5. 2019Deep Learning for 3D Point Clouds: A Surveycited by 558
  6. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  7. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 504
  8. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  9. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  10. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 423
  11. 2020Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentationcited by 389
  12. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  13. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 300
  14. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 246
  15. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  16. 2021Spatio-temporal Self-Supervised Representation Learning for 3D Point Cloudscited by 180
  17. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  18. 2021RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentationcited by 173
  19. 2014Evaluation of Output Embeddings for Fine-Grained Image Classificationcited by 166
  20. 2021Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Trackingcited by 166
  21. 2021(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Networkcited by 164
  22. 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 158
  23. 2017Semantic Autoencoder for Zero-Shot Learningcited by 133
  24. 2016Latent Embeddings for Zero-shot Classificationcited by 129
  25. 2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
  26. 2022Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentationcited by 119
  27. 2016Synthesized Classifiers for Zero-Shot Learningcited by 113
  28. 2022LaserMix for Semi-Supervised LiDAR Semantic Segmentationcited by 93
  29. 2017A Generative Model For Zero Shot Learning Using Conditional Variational Autoencoderscited by 64
  30. 2021Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Cloudscited by 55
  31. 2021ConDA: Unsupervised Domain Adaptation for LiDAR Segmentation via Regularized Domain Concatenationcited by 41
  32. 2020Unsupervised Learning of Intrinsic Structural Representation Pointscited by 39
  33. 2021Zero-Shot Learning on 3D Point Cloud Objects and Beyondcited by 38
  34. 2017Generating Visual Representations for Zero-Shot Classificationcited by 37
  35. 2019Zero-shot Learning of 3D Point Cloud Objectscited by 34
  36. 2021Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretrainingcited by 33
  37. 2019Mitigating the Hubness Problem for Zero-Shot Learning of 3D Objectscited by 24
  38. 2017Attributes2Classname: A discriminative model for attribute-based unsupervised zero-shot learningcited by 12

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 36; 2024: 57; 2025: 62; 2026: 26.

181 papers.

Sort:

By area:

  1. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 184
  2. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  3. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
  4. 2023Rethinking Range View Representation for LiDAR Segmentationcited by 128
  5. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  6. 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 125
  7. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  8. 2024SAMPart3D: Segment Any Part in 3D Objectscited by 97
  9. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  10. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  11. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  12. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 67
38 more
  1. 2023Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labelscited by 64
  2. 2023UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebasecited by 57
  3. 2024IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detectioncited by 57
  4. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 57
  5. 2023Towards Label-free Scene Understanding by Vision Foundation Modelscited by 54
  6. 2024Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Drivingcited by 54
  7. 2025Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Modelscited by 54
  8. 2023FRNet: Frustum-Range Networks for Scalable LiDAR Segmentationcited by 50
  9. 2025Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representationscited by 45
  10. 2023Three Pillars improving Vision Foundation Model Distillation for Lidarcited by 44
  11. 2023SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentationcited by 44
  12. 2024Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learningcited by 43
  13. 2024The Double-Ellipsoid Geometry of CLIPcited by 42
  14. 2024Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Drivingcited by 41
  15. 2024Few-Shot Adversarial Prompt Learning on Vision-Language Modelscited by 39
  16. 2023Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understandingcited by 38
  17. 2023Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Drivingcited by 37
  18. 20244D Contrastive Superflows are Dense 3D Representation Learnerscited by 33
  19. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 33
  20. 2023Label-Free Event-based Object Recognition via Joint Learning with Image Reconstruction from Eventscited by 28
  21. 2023PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillationcited by 28
  22. 2023Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation Modelscited by 27
  23. 2024Is Your LiDAR Placement Optimized for 3D Scene Understanding?cited by 26
  24. 2023TENT: Connect Language Models with IoT Sensors for Zero-Shot Activity Recognitioncited by 25
  25. 2024Multi-Space Alignments Towards Universal LiDAR Segmentationcited by 25
  26. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  27. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 23
  28. 2023See More and Know More: Zero-shot Point Cloud Segmentation via Multi-modal Visual Datacited by 21
  29. 2024Find n' Propagate: Open-Vocabulary 3D Object Detection in Urban Environmentscited by 21
  30. 2023A Survey of Label-Efficient Deep Learning for 3D Point Cloudscited by 20
  31. 2023Learning to Adapt SAM for Segmenting Cross-domain Point Cloudscited by 20
  32. 2024OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of Allcited by 20
  33. 2024Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentationcited by 20
  34. 2024PartSTAD: 2D-to-3D Part Segmentation Task Adaptationcited by 18
  35. 2024GAGS: Granularity-Aware Feature Distillation for Language Gaussian Splattingcited by 18
  36. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Modelscited by 18
  37. 2024Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillationcited by 17
  38. 2025Generalized Few-shot 3D Point Cloud Segmentation with Vision-Language Modelcited by 17

50 most cited of 181 All 181 (JSON)