# Geometrically-driven Aggregation for Zero-shot 3D Point Cloud Understanding

arXiv: https://arxiv.org/abs/2312.02244 · HTML: https://mlsanity.com/paper/2312.02244/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 33 of its 40 references that are in the map.
- First posted on arXiv 4 Dec 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data](https://mlsanity.com/paper/1908.04616/) (2019)
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [Point Transformer V2: Grouped Vector Attention and Partition-based Pooling](https://mlsanity.com/paper/2210.05666/) (2022)
- [Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs](https://mlsanity.com/paper/1711.09869/) (2017)
- [Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework](https://mlsanity.com/paper/2202.07123/) (2022)
- [Relation-Shape Convolutional Neural Network for Point Cloud Analysis](https://mlsanity.com/paper/1904.07601/) (2019)
- [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023)
- [ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding](https://mlsanity.com/paper/2212.05171/) (2022)
- [PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning](https://mlsanity.com/paper/2211.11682/) (2022)
- [CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-training](https://mlsanity.com/paper/2210.01055/) (2022)
- [Fast Point Transformer](https://mlsanity.com/paper/2112.04702/) (2021)
- [CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition](https://mlsanity.com/paper/2303.11313/) (2023)
- [CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data](https://mlsanity.com/paper/2303.12417/) (2023)
- [Recurrent Pixel Embedding for Instance Grouping](https://mlsanity.com/paper/1712.08273/) (2017)
- [Single-Image Piece-wise Planar 3D Reconstruction via Associative Embedding](https://mlsanity.com/paper/1902.09777/) (2019)
- [VConv-DAE: Deep Volumetric Shape Learning Without Object Labels](https://mlsanity.com/paper/1604.03755/) (2016)
- [Zero-Shot Learning on 3D Point Cloud Objects and Beyond](https://mlsanity.com/paper/2104.04980/) (2021)

## Built on by (most cited first, 13 of 13)

- [Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2410.22489/) (2024)
- [Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2503.16707/) (2025)
- [Vocabulary-Free 3D Instance Segmentation with Vision and Language Assistant](https://mlsanity.com/paper/2408.10652/) (2024)
- [GSTran: Joint Geometric and Semantic Coherence for Point Cloud Segmentation](https://mlsanity.com/paper/2408.11558/) (2024)
- [Parameter-Efficient CLIP Adaptation for 3D Understanding via Unified Tokenization](https://mlsanity.com/paper/2505.18819/) (2025)
- [GLASS: Graph and Vision-Language Assisted Semantic Shape Correspondence](https://mlsanity.com/paper/2603.07652/) (2026)
- [GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation](https://mlsanity.com/paper/2603.26260/) (2026)
- [Exploiting Vision Language Model for Training-Free 3D Point Cloud OOD Detection via Graph Score Propagation](https://mlsanity.com/paper/2506.22375/) (2025)
- [Real-Time 3D Object Detection with Inference-Aligned Learning](https://mlsanity.com/paper/2511.16140/) (2025)
- [SegGraph: Leveraging Graphs of SAM Segments for Few-Shot 3D Part Segmentation](https://mlsanity.com/paper/2512.16143/) (2025)
- [Data-Efficient Semantic Segmentation of 3D Point Clouds via Open-Vocabulary Image Segmentation-based Pseudo-Labeling](https://mlsanity.com/paper/2604.11007/) (2026)
- [MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshes](https://mlsanity.com/paper/2606.31533/) (2026)
- [xperception -- Making Robotic Grasping Easier](https://mlsanity.com/paper/2607.16312/) (2026)

## Abstract (arXiv)

Zero-shot 3D point cloud understanding can be achieved via 2D Vision-Language Models (VLMs). Existing strategies directly map Vision-Language Models from 2D pixels of rendered or captured views to 3D points, overlooking the inherent and expressible point cloud geometric structure. Geometrically similar or close regions can be exploited for bolstering point cloud understanding as they are likely to share semantic information. To this end, we introduce the first training-free aggregation technique that leverages the point cloud's 3D geometric structure to improve the quality of the transferred Vision-Language Models. Our approach operates iteratively, performing local-to-global aggregation based on geometric and semantic point-level reasoning. We benchmark our approach on three downstream tasks, including classification, part segmentation, and semantic segmentation, with a variety of datasets representing both synthetic/real-world, and indoor/outdoor scenarios. Our approach achieves new state-of-the-art results in all benchmarks. Our approach operates iteratively, performing local-to-global aggregation based on geometric and semantic point-level reasoning. Code and dataset are available at https://luigiriz.github.io/geoze-website/