# Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds

arXiv: https://arxiv.org/abs/2108.06230 · HTML: https://mlsanity.com/paper/2108.06230/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 86 references that are in the map.
- First posted on arXiv 13 Aug 2021.

## Builds on (most cited first, 30 of 61)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs](https://mlsanity.com/paper/1711.09869/) (2017)
- [Adversarial Autoencoders](https://mlsanity.com/paper/1511.05644/) (2015)
- [Relation-Shape Convolutional Neural Network for Point Cloud Analysis](https://mlsanity.com/paper/1904.07601/) (2019)
- [OctNet: Learning Deep 3D Representations at High Resolutions](https://mlsanity.com/paper/1611.05009/) (2016)
- [SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters](https://mlsanity.com/paper/1803.11527/) (2018)
- [Generative Moment Matching Networks](https://mlsanity.com/paper/1502.02761/) (2015)
- [Feature Generating Networks for Zero-Shot Learning](https://mlsanity.com/paper/1712.00981/) (2017)

## Built on by (most cited first, 30 of 55)

- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [Language-Grounded Indoor 3D Semantic Segmentation in the Wild](https://mlsanity.com/paper/2204.07761/) (2022)
- [PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning](https://mlsanity.com/paper/2211.11682/) (2022)
- [CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP](https://mlsanity.com/paper/2301.04926/) (2023)
- [PLA: Language-Driven Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2211.16312/) (2022)
- [Segment Any Point Cloud Sequences by Distilling Vision Foundation Models](https://mlsanity.com/paper/2306.09347/) (2023)
- [RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding](https://mlsanity.com/paper/2304.00962/) (2023)
- [CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP](https://mlsanity.com/paper/2303.04748/) (2023)
- [OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation](https://mlsanity.com/paper/2309.00616/) (2023)
- [SATR: Zero-Shot Semantic Segmentation of 3D Shapes](https://mlsanity.com/paper/2304.04909/) (2023)
- [A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future](https://mlsanity.com/paper/2307.09220/) (2023)
- [Towards Label-free Scene Understanding by Vision Foundation Models](https://mlsanity.com/paper/2306.03899/) (2023)
- [SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentation](https://mlsanity.com/paper/2311.17707/) (2023)
- [Open-Vocabulary Affordance Detection in 3D Point Clouds](https://mlsanity.com/paper/2303.02401/) (2023)
- [Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understanding](https://mlsanity.com/paper/2308.00353/) (2023)
- [Segment Any Mesh](https://mlsanity.com/paper/2408.13679/) (2024)
- [LidarCLIP or: How I Learned to Talk to Point Clouds](https://mlsanity.com/paper/2212.06858/) (2022)
- [Zero-Shot 3D Shape Correspondence](https://mlsanity.com/paper/2306.03253/) (2023)
- [3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds](https://mlsanity.com/paper/2502.20041/) (2025)
- [Prototype Adaption and Projection for Few- and Zero-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2305.14335/) (2023)
- [Open-Vocabulary 3D Detection via Image-level Class and Debiased Cross-modal Contrastive Learning](https://mlsanity.com/paper/2207.01987/) (2022)
- [OpenSight: A Simple Open-Vocabulary Framework for LiDAR-Based Object Detection](https://mlsanity.com/paper/2312.08876/) (2023)
- [MeshSegmenter: Zero-Shot Mesh Semantic Segmentation via Texture Synthesis](https://mlsanity.com/paper/2407.13675/) (2024)
- [See More and Know More: Zero-shot Point Cloud Segmentation via Multi-modal Visual Data](https://mlsanity.com/paper/2307.10782/) (2023)
- [A Survey of Label-Efficient Deep Learning for 3D Point Clouds](https://mlsanity.com/paper/2305.19812/) (2023)
- [3D Compositional Zero-shot Learning with DeCompositional Consensus](https://mlsanity.com/paper/2111.14673/) (2021)
- [Visual Semantic Segmentation Based on Few/Zero-Shot Learning: An Overview](https://mlsanity.com/paper/2211.08352/) (2022)
- [UniM-OV3D: Uni-Modality Open-Vocabulary 3D Scene Understanding with Fine-Grained Feature Representation](https://mlsanity.com/paper/2401.11395/) (2024)
- [Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2407.13642/) (2024)

## Abstract (arXiv)

While there has been a number of studies on Zero-Shot Learning (ZSL) for 2D images, its application to 3D data is still recent and scarce, with just a few methods limited to classification. We present the first generative approach for both ZSL and Generalized ZSL (GZSL) on 3D data, that can handle both classification and, for the first time, semantic segmentation. We show that it reaches or outperforms the state of the art on ModelNet40 classification for both inductive ZSL and inductive GZSL. For semantic segmentation, we created three benchmarks for evaluating this new ZSL task, using S3DIS, ScanNet and SemanticKITTI. Our experiments show that our method outperforms strong baselines, which we additionally propose for this task.