mlsanityData to 7 Oct 2026

Geometrically-driven Aggregation for Zero-shot 3D Point Cloud Understanding

arXiv 2312.02244 ↗First posted 4 Dec 2023cs.CVcited by 13

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Zero-shot 3D point cloud understanding can be achieved via 2D Vision-Language Models (VLMs). Existing strategies directly map Vision-Language Models from 2D pixels of rendered or captured views to 3D points, overlooking the inherent and expressible point cloud geometric structure. Geometrically similar or close regions can be exploited for bolstering point cloud understanding as they are likely to share semantic information. To this end, we introduce the first training-free aggregation technique that leverages the point cloud's 3D geometric structure to improve the quality of the transferred Vision-Language Models. Our approach operates iteratively, performing local-to-global aggregation based on geometric and semantic point-level reasoning. We benchmark our approach on three downstream tasks, including classification, part segmentation, and semantic segmentation, with a variety of datasets representing both synthetic/real-world, and indoor/outdoor scenarios. Our approach achieves new state-of-the-art results in all benchmarks. Our approach operates iteratively, performing local-to-global aggregation based on geometric and semantic point-level reasoning. Code and dataset are available at https://luigiriz.github.io/geoze-website/

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 40 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

33 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition20 Mar
  2. 2021PointCLIP: Point Cloud Understanding by CLIP4 Dec

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  6. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  8. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  9. 2020Point Transformercited by 1,462
  10. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  11. 2022Language-driven Semantic Segmentationcited by 711
  12. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
19 more
  1. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  2. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  3. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 384
  4. 2022Point Transformer V2: Grouped Vector Attention and Partition-based Poolingcited by 384
  5. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 376
  6. 2019Relation-Shape Convolutional Neural Network for Point Cloud Analysiscited by 346
  7. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  8. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  9. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  10. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  11. 2021Fast Point Transformercited by 103
  12. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69
  13. 2017Recurrent Pixel Embedding for Instance Groupingcited by 57
  14. 2019Single-Image Piece-wise Planar 3D Reconstruction via Associative Embeddingcited by 56
  15. 2016VConv-DAE: Deep Volumetric Shape Learning Without Object Labelscited by 50
  16. 2021Zero-Shot Learning on 3D Point Cloud Objects and Beyondcited by 38
  17. 2023Unsupervised Deep Probabilistic Approach for Partial Point Cloud Registrationcited by 26
  18. 2019Mitigating the Hubness Problem for Zero-Shot Learning of 3D Objectscited by 24
  19. 2020PanoNet3D: Combining Semantic and Geometric Understanding for LiDARPoint Cloud Detectioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 5.

13 papers.

Sort:

By area:

  1. 2024Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentationcited by 20
  2. 2025Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understandingcited by 13
  3. 2024Vocabulary-Free 3D Instance Segmentation with Vision and Language Assistantcited by 5
  4. 2024GSTran: Joint Geometric and Semantic Coherence for Point Cloud Segmentationcited by 1
  5. 2025Parameter-Efficient CLIP Adaptation for 3D Understanding via Unified Tokenizationcited by 1
  6. 2026GLASS: Graph and Vision-Language Assisted Semantic Shape Correspondencecited by 1
  7. 2026GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentationcited by 1
  8. 2025Exploiting Vision Language Model for Training-Free 3D Point Cloud OOD Detection via Graph Score Propagationcited by 0
  9. 2025Real-Time 3D Object Detection with Inference-Aligned Learningcited by 0
  10. 2025SegGraph: Leveraging Graphs of SAM Segments for Few-Shot 3D Part Segmentationcited by 0
  11. 2026Data-Efficient Semantic Segmentation of 3D Point Clouds via Open-Vocabulary Image Segmentation-based Pseudo-Labelingcited by 0
  12. 2026MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshescited by 0
1 more
  1. 2026xperception -- Making Robotic Grasping Easiercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20253D-PointZshotS: Geometry-Aware 3D Point Cloud Zero-Shot Semantic Segmentation Narrowing the Visual-Semantic Gap
  2. 2025Point-MoE: Large-Scale Multi-Dataset Training with Mixture-of-Experts for 3D Semantic Segmentation
  3. 2026PointLLM-R: Enhancing 3D Point Cloud Reasoning via Chain-of-Thought
  4. 2026Leveraging 2D-VLM for Label-Free 3D Segmentation in Large-Scale Outdoor Scene Understanding
  5. 2026Point Cloud as a Foreign Language for Multi-modal Large Language Model
  6. 2024ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models
  7. 2025SODA: Out-of-Distribution Detection in Domain-Shifted Point Clouds via Neighborhood Propagation
  8. 2026SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes