# Generalized Few-shot 3D Point Cloud Segmentation with Vision-Language Model

arXiv: https://arxiv.org/abs/2503.16282 · HTML: https://mlsanity.com/paper/2503.16282/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 61 of its 82 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 61)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021)
- [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023)
- [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023)
- [Language-Grounded Indoor 3D Semantic Segmentation in the Wild](https://mlsanity.com/paper/2204.07761/) (2022)
- [CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP](https://mlsanity.com/paper/2301.04926/) (2023)
- [PLA: Language-Driven Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2211.16312/) (2022)
- [DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection](https://mlsanity.com/paper/2209.09407/) (2022)
- [CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-training](https://mlsanity.com/paper/2210.01055/) (2022)
- [Mix3D: Out-of-Context Data Augmentation for 3D Scenes](https://mlsanity.com/paper/2110.02210/) (2021)
- [Learning What Not to Segment: A New Perspective on Few-Shot Segmentation](https://mlsanity.com/paper/2203.07615/) (2022)
- [PointMixup: Augmentation for Point Clouds](https://mlsanity.com/paper/2008.06374/) (2020)
- [Fast Point Transformer](https://mlsanity.com/paper/2112.04702/) (2021)
- [OneFormer3D: One Transformer for Unified Point Cloud Segmentation](https://mlsanity.com/paper/2311.14405/) (2023)
- [RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding](https://mlsanity.com/paper/2304.00962/) (2023)
- [CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP](https://mlsanity.com/paper/2303.04748/) (2023)
- [Few-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2006.12052/) (2020)
- [PolarMix: A General Data Augmentation Technique for LiDAR Point Clouds](https://mlsanity.com/paper/2208.00223/) (2022)
- [Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detection](https://mlsanity.com/paper/2406.10700/) (2024)
- [PointAugment: an Auto-Augmentation Framework for Point Cloud Classification](https://mlsanity.com/paper/2002.10876/) (2020)
- [Generalized Few-shot Semantic Segmentation](https://mlsanity.com/paper/2010.05210/) (2020)

## Built on by (most cited first, 17 of 17)

- [Domain-RAG: Retrieval-Guided Compositional Image Generation for Cross-Domain Few-Shot Object Detection](https://mlsanity.com/paper/2506.05872/) (2025)
- [RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding](https://mlsanity.com/paper/2505.14462/) (2025)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models](https://mlsanity.com/paper/2603.09826/) (2026)
- [Few-Shot Learning in Video and 3D Object Detection: A Survey](https://mlsanity.com/paper/2507.17079/) (2025)
- [Novel Class Discovery for Point Cloud Segmentation via Joint Learning of Causal Representation and Reasoning](https://mlsanity.com/paper/2510.13307/) (2025)
- [Synthesizing High-Quality Visual Question Answering from Medical Documents with Generator-Verifier LMMs](https://mlsanity.com/paper/2510.25867/) (2025)
- [Few-Shot Incremental 3D Object Detection in Dynamic Indoor Environments](https://mlsanity.com/paper/2604.07997/) (2026)
- [Learning Hierarchical Orthogonal Prototypes for Generalized Few-Shot 3D Point Cloud Segmentation](https://mlsanity.com/paper/2603.19788/) (2026)
- [Make It Up: Fake Images, Real Gains in Generalized Few-shot Semantic Segmentation](https://mlsanity.com/paper/2603.27206/) (2026)
- [THOM: Generating Physically Plausible Hand-Object Meshes From Text](https://mlsanity.com/paper/2604.02736/) (2026)
- [Unsupervised Online 3D Instance Segmentation with Synthetic Sequences and Dynamic Loss](https://mlsanity.com/paper/2509.23194/) (2025)
- [Beyond Visual Cues: Leveraging General Semantics as Support for Few-Shot Segmentation](https://mlsanity.com/paper/2511.16435/) (2025)
- [Rethinking Multimodal Few-Shot 3D Point Cloud Segmentation: From Fused Refinement to Decoupled Arbitration](https://mlsanity.com/paper/2601.01456/) (2026)
- [UniD-Shift: Towards Unified Semantic Segmentation via Interpretable Share-Private Multimodal Decomposition](https://mlsanity.com/paper/2605.07356/) (2026)
- [Training-Free Open-Vocabulary 3D Point-Cloud Segmentation on the Generalized Few-Shot Benchmark](https://mlsanity.com/paper/2607.15331/) (2026)
- [Beyond Argmax: A Mechanistic Study of Semantic Retention in Frozen Foundation-Model Composition for Generalized Few-Shot 3D Segmentation](https://mlsanity.com/paper/2609.12099/) (2026)

## Abstract (arXiv)

Generalized few-shot 3D point cloud segmentation (GFS-PCS) adapts models to new classes with few support samples while retaining base class segmentation. Existing GFS-PCS methods enhance prototypes via interacting with support or query features but remain limited by sparse knowledge from few-shot samples. Meanwhile, 3D vision-language models (3D VLMs), generalizing across open-world novel classes, contain rich but noisy novel class knowledge. In this work, we introduce a GFS-PCS framework that synergizes dense but noisy pseudo-labels from 3D VLMs with precise yet sparse few-shot samples to maximize the strengths of both, named GFS-VL. Specifically, we present a prototype-guided pseudo-label selection to filter low-quality regions, followed by an adaptive infilling strategy that combines knowledge from pseudo-label contexts and few-shot samples to adaptively label the filtered, unlabeled areas. Additionally, we design a novel-base mix strategy to embed few-shot samples into training scenes, preserving essential context for improved novel class learning. Moreover, recognizing the limited diversity in current GFS-PCS benchmarks, we introduce two challenging benchmarks with diverse novel classes for comprehensive generalization evaluation. Experiments validate the effectiveness of our framework across models and datasets. Our approach and benchmarks provide a solid foundation for advancing GFS-PCS in the real world. The code is at https://github.com/ZhaochongAn/GFS-VL