# Zero-Shot Learning on 3D Point Cloud Objects and Beyond

arXiv: https://arxiv.org/abs/2104.04980 · HTML: https://mlsanity.com/paper/2104.04980/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 81 references that are in the map.
- First posted on arXiv 11 Apr 2021.

## Builds on (most cited first, 30 of 55)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data](https://mlsanity.com/paper/1908.04616/) (2019)
- [SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters](https://mlsanity.com/paper/1803.11527/) (2018)
- [Feature Generating Networks for Zero-Shot Learning](https://mlsanity.com/paper/1712.00981/) (2017)
- [SO-Net: Self-Organizing Network for Point Cloud Analysis](https://mlsanity.com/paper/1803.04249/) (2018)
- [Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders](https://mlsanity.com/paper/1812.01784/) (2018)
- [Evaluation of Output Embeddings for Fine-Grained Image Classification](https://mlsanity.com/paper/1409.8403/) (2014)
- [Label-Embedding for Image Classification](https://mlsanity.com/paper/1503.08677/) (2015)
- [f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning](https://mlsanity.com/paper/1903.10132/) (2019)
- [Learning a Deep Embedding Model for Zero-Shot Learning](https://mlsanity.com/paper/1611.05088/) (2016)
- [An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild](https://mlsanity.com/paper/1605.04253/) (2016)
- [Latent Embeddings for Zero-shot Classification](https://mlsanity.com/paper/1603.08895/) (2016)
- [Synthesized Classifiers for Zero-Shot Learning](https://mlsanity.com/paper/1603.00550/) (2016)
- [PointMixup: Augmentation for Point Clouds](https://mlsanity.com/paper/2008.06374/) (2020)
- [Local Spectral Graph Convolution for Point Set Feature Learning](https://mlsanity.com/paper/1803.05827/) (2018)
- [Leveraging the Invariant Side of Generative Zero-Shot Learning](https://mlsanity.com/paper/1904.04092/) (2019)
- [Multi-Label Zero-Shot Learning with Structured Knowledge Graphs](https://mlsanity.com/paper/1711.06526/) (2017)
- [Triplet-Center Loss for Multi-View 3D Object Retrieval](https://mlsanity.com/paper/1803.06189/) (2018)

## Built on by (most cited first, 30 of 38)

- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [Language-Grounded Indoor 3D Semantic Segmentation in the Wild](https://mlsanity.com/paper/2204.07761/) (2022)
- [PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning](https://mlsanity.com/paper/2211.11682/) (2022)
- [CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP](https://mlsanity.com/paper/2301.04926/) (2023)
- [CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-training](https://mlsanity.com/paper/2210.01055/) (2022)
- [CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP](https://mlsanity.com/paper/2303.04748/) (2023)
- [CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data](https://mlsanity.com/paper/2303.12417/) (2023)
- [Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/2108.06230/) (2021)
- [LidarCLIP or: How I Learned to Talk to Point Clouds](https://mlsanity.com/paper/2212.06858/) (2022)
- [PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection](https://mlsanity.com/paper/2410.00320/) (2024)
- [PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillation](https://mlsanity.com/paper/2312.04016/) (2023)
- [Open-Vocabulary 3D Detection via Image-level Class and Debiased Cross-modal Contrastive Learning](https://mlsanity.com/paper/2207.01987/) (2022)
- [OpenSight: A Simple Open-Vocabulary Framework for LiDAR-Based Object Detection](https://mlsanity.com/paper/2312.08876/) (2023)
- [See More and Know More: Zero-shot Point Cloud Segmentation via Multi-modal Visual Data](https://mlsanity.com/paper/2307.10782/) (2023)
- [Few-shot Class-incremental Learning for 3D Point Cloud Objects](https://mlsanity.com/paper/2205.15225/) (2022)
- [Geometrically-driven Aggregation for Zero-shot 3D Point Cloud Understanding](https://mlsanity.com/paper/2312.02244/) (2023)
- [Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2407.13642/) (2024)
- [MV-CLIP: Multi-View CLIP for Zero-shot 3D Shape Recognition](https://mlsanity.com/paper/2311.18402/) (2023)
- [Zero-shot point cloud segmentation by transferring geometric primitives](https://mlsanity.com/paper/2210.09923/) (2022)
- [Towards Zero-shot 3D Anomaly Localization](https://mlsanity.com/paper/2412.04304/) (2024)
- [Learning without Forgetting for 3D Point Cloud Objects](https://mlsanity.com/paper/2106.14275/) (2021)
- [Reliability-Adaptive Consistency Regularization for Weakly-Supervised Point Cloud Segmentation](https://mlsanity.com/paper/2303.05164/) (2023)
- [Exploiting GPT-4 Vision for Zero-shot Point Cloud Understanding](https://mlsanity.com/paper/2401.07572/) (2024)
- [On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Survey](https://mlsanity.com/paper/2408.04879/) (2024)
- [Open-Pose 3D Zero-Shot Learning: Benchmark and Challenges](https://mlsanity.com/paper/2312.07039/) (2023)
- [ExpPoint-MAE: Better interpretability and performance for self-supervised point cloud transformers](https://mlsanity.com/paper/2306.10798/) (2023)
- [Prompt-guided Scene Generation for 3D Zero-Shot Learning](https://mlsanity.com/paper/2209.14690/) (2022)
- [Generalized Zero-Shot Learning for Point Cloud Segmentation with Evidence-Based Dynamic Calibration](https://mlsanity.com/paper/2509.08280/) (2025)
- [MCL-AD: Multimodal Collaboration Learning for Zero-Shot 3D Anomaly Detection](https://mlsanity.com/paper/2509.10282/) (2025)

## Abstract (arXiv)

Zero-shot learning, the task of learning to recognize new classes not seen during training, has received considerable attention in the case of 2D image classification. However, despite the increasing ubiquity of 3D sensors, the corresponding 3D point cloud classification problem has not been meaningfully explored and introduces new challenges. In this paper, we identify some of the challenges and apply 2D Zero-Shot Learning (ZSL) methods in the 3D domain to analyze the performance of existing models. Then, we propose a novel approach to address the issues specific to 3D ZSL. We first present an inductive ZSL process and then extend it to the transductive ZSL and Generalized ZSL (GZSL) settings for 3D point cloud classification. To this end, a novel loss function is developed that simultaneously aligns seen semantics with point cloud features and takes advantage of unlabeled test data to address some known issues (e.g., the problems of domain adaptation, hubness, and data bias). While designed for the particularities of 3D point cloud classification, the method is shown to also be applicable to the more common use-case of 2D image classification. An extensive set of experiments is carried out, establishing state-of-the-art for ZSL and GZSL on synthetic (ModelNet40, ModelNet10, McGill) and real (ScanObjectNN) 3D point cloud datasets.