mlsanityData to 7 Oct 2026

See More and Know More: Zero-shot Point Cloud Segmentation via Multi-modal Visual Data

arXiv 2307.10782 ↗First posted 20 Jul 2023cs.CVcited by 21

Not on the map. Why?

Abstract (arXiv)

Zero-shot point cloud segmentation aims to make deep models capable of recognizing novel objects in point cloud that are unseen in the training phase. Recent trends favor the pipeline which transfers knowledge from seen classes with labels to unseen classes without labels. They typically align visual features with semantic features obtained from word embedding by the supervision of seen classes' annotations. However, point cloud contains limited information to fully match with semantic features. In fact, the rich appearance information of images is a natural complement to the textureless point cloud, which is not well explored in previous literature. Motivated by this, we propose a novel multi-modal zero-shot learning method to better utilize the complementary information of point clouds and images for more accurate visual-semantic alignment. Extensive experiments are performed in two popular benchmarks, i.e., SemanticKITTI and nuScenes, and our method outperforms current SOTA methods with 52% and 49% improvement on average for unseen class mIoU, respectively.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 54 of its 76 references that are in the map. First posted on arXiv 20 Jul 2023.

Builds on

54 of 76 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2022Language-driven Semantic Segmentationcited by 711
  8. 2021Extract Free Dense Labels from CLIPcited by 580
  9. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  10. 2021Image Segmentation Using Text and Image Promptscited by 509
  11. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  12. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 464
38 more
  1. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  2. 2020Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentationcited by 388
  3. 2022BEVFusion: A Simple and Robust LiDAR-Camera Fusion Frameworkcited by 334
  4. 2019Zero-Shot Semantic Segmentationcited by 245
  5. 2022DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detectioncited by 224
  6. 2021Decoupling Zero-Shot Semantic Segmentationcited by 198
  7. 2022FUTR3D: A Unified Sensor Fusion Framework for 3D Detectioncited by 194
  8. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  9. 2021RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentationcited by 173
  10. 2017PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimationcited by 170
  11. 2014Evaluation of Output Embeddings for Fine-Grained Image Classificationcited by 166
  12. 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 157
  13. 2016An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wildcited by 146
  14. 2013Zero-Shot Learning by Convex Combination of Semantic Embeddingscited by 135
  15. 2017Semantic Autoencoder for Zero-Shot Learningcited by 133
  16. 2016Latent Embeddings for Zero-shot Classificationcited by 129
  17. 2023Rethinking Range View Representation for LiDAR Segmentationcited by 126
  18. 2016Synthesized Classifiers for Zero-Shot Learningcited by 113
  19. 2020Context-aware Feature Generation for Zero-shot Semantic Segmentationcited by 94
  20. 2020LiDAR-based Panoptic Segmentation via Dynamic Shifting Networkcited by 84
  21. 2022Video Polyp Segmentation: A Deep Learning Perspectivecited by 81
  22. 2015Zero-Shot Learning via Semantic Similarity Embeddingcited by 74
  23. 2017A Generative Model For Zero Shot Learning Using Conditional Variational Autoencoderscited by 64
  24. 2021EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentationcited by 59
  25. 2021A Closer Look at Self-training for Zero-Label Semantic Segmentationcited by 57
  26. 2021Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Cloudscited by 55
  27. 2023Towards Label-free Scene Understanding by Vision Foundation Modelscited by 54
  28. 2021Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentationcited by 53
  29. 2018Transductive Unbiased Embedding for Zero-Shot Learningcited by 44
  30. 2021SIGN: Spatial-information Incorporated Generative Network for Generalized Zero-shot Semantic Segmentationcited by 41
  31. 2021Zero-Shot Instance Segmentationcited by 40
  32. 2019Transductive Zero-Shot Learning for 3D Point Cloud Classificationcited by 39
  33. 2021Zero-Shot Learning on 3D Point Cloud Objects and Beyondcited by 38
  34. 2017Generating Visual Representations for Zero-Shot Classificationcited by 37
  35. 2019Zero-shot Learning of 3D Point Cloud Objectscited by 34
  36. 2019Mitigating the Hubness Problem for Zero-Shot Learning of 3D Objectscited by 24
  37. 2017Attributes2Classname: A discriminative model for attribute-based unsupervised zero-shot learningcited by 12
  38. 2020Learning unbiased zero-shot semantic segmentation networks via transductive transfercited by 11

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 11; 2025: 7; 2026: 1.

21 papers.

Sort:

By area:

  1. 2023UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebasecited by 57
  2. 2024IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detectioncited by 57
  3. 2024Better Call SAL: Towards Learning to Segment Anything in Lidarcited by 47
  4. 2024Multi-Space Alignments Towards Universal LiDAR Segmentationcited by 24
  5. 2024LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation in Free Environmentcited by 20
  6. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentationcited by 16
  7. 2024Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for Scene Flowcited by 15
  8. 2025SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenescited by 11
  9. 2024ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Drivingcited by 6
  10. 2025Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systemscited by 4
  11. 2023Model2Scene: Learning 3D Scene Representation via Contrastive Language-CAD Models Pre-trainingcited by 3
  12. 2024PanoSLAM: Panoptic 3D Scene Reconstruction via Gaussian SLAMcited by 3
9 more
  1. 2025SAM4D: Segment Anything in Camera and LiDAR Streamscited by 3
  2. 2024HUNTER: Unsupervised Human-centric 3D Detection via Transferring Knowledge from Synthetic Instances to Real Scenescited by 2
  3. 2024PRENet: A Plane-Fit Redundancy Encoding Point Cloud Sequence Network for Real-Time 3D Action Recognitioncited by 2
  4. 2024Towards Practical Human Motion Prediction with LiDAR Point Cloudscited by 2
  5. 2025Generalized Zero-Shot Learning for Point Cloud Segmentation with Evidence-Based Dynamic Calibrationcited by 2
  6. 2024Triple Point Maskingcited by 1
  7. 20253D Can Be Explored In 2D: Pseudo-Label Generation for LiDAR Point Clouds Using Sensor-Intensity-Based 2D Semantic Segmentationcited by 1
  8. 2025GZSL-MoE: Apprentissage G'en'eralis'e Z'ero-Shot bas'e sur le M'elange d'Experts pour la Segmentation S'emantique de Nuages de Points 3DAppliqu'e `a un Jeu de Donn'ees d'Environnement de Collaboration Humain-Robotcited by 0
  9. 2026XMatchAD: A Cross-Modal Matching Perspective on Reconstruction-based Anomaly Detectioncited by 0