# PGOV3D: Open-Vocabulary 3D Semantic Segmentation with Partial-to-Global Curriculum

arXiv: https://arxiv.org/abs/2506.23607 · HTML: https://mlsanity.com/paper/2506.23607/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 53 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Builds on (most cited first, 30 of 42)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [YOLO-World: Real-Time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2401.17270/) (2024)
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [Side Adapter Network for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2302.12242/) (2023)
- [Language-Grounded Indoor 3D Semantic Segmentation in the Wild](https://mlsanity.com/paper/2204.07761/) (2022)
- [SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2203.01509/) (2022)
- [PLA: Language-Driven Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2211.16312/) (2022)
- [MSeg: A Composite Dataset for Multi-domain Semantic Segmentation](https://mlsanity.com/paper/2112.13762/) (2021)
- [Superpoint Transformer for 3D Scene Instance Segmentation](https://mlsanity.com/paper/2211.15766/) (2022)
- [OneFormer3D: One Transformer for Unified Point Cloud Segmentation](https://mlsanity.com/paper/2311.14405/) (2023)
- [RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding](https://mlsanity.com/paper/2304.00962/) (2023)
- [OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation](https://mlsanity.com/paper/2309.00616/) (2023)

## Built on by (most cited first, 4 of 4)

- [GeoPurify: A Data-Efficient Geometric Distillation Framework for Open-Vocabulary 3D Segmentation](https://mlsanity.com/paper/2510.02186/) (2025)
- [GeoGuide: Hierarchical Geometric Guidance for Open-Vocabulary 3D Semantic Segmentation](https://mlsanity.com/paper/2603.26260/) (2026)
- [Lightweight 3D Feature Pretraining by Bayesian Inversion of 2D Foundation Models](https://mlsanity.com/paper/2606.21292/) (2026)
- [Lift, Associate, and Fuse: A Decision-Centric Framework for 2D-to-3D Foundation Model Transfer](https://mlsanity.com/paper/2608.20659/) (2026)

## Abstract (arXiv)

Existing open-vocabulary 3D semantic segmentation methods typically supervise 3D segmentation models by merging text-aligned features (e.g., CLIP) extracted from multi-view images onto 3D points. However, such approaches treat multi-view images merely as intermediaries for transferring open-vocabulary information, overlooking their rich semantic content and cross-view correspondences, which limits model effectiveness. To address this, we propose PGOV3D, a novel framework that introduces a Partial-to-Global curriculum for improving open-vocabulary 3D semantic segmentation. The key innovation lies in a two-stage training strategy. In the first stage, we pre-train the model on partial scenes that provide dense semantic information but relatively simple geometry. These partial point clouds are derived from multi-view RGB-D inputs via pixel-wise depth projection. To enable open-vocabulary learning, we leverage a multi-modal large language model (MLLM) and a 2D segmentation foundation model to generate open-vocabulary labels for each viewpoint, offering rich and aligned supervision. An auxiliary inter-frame consistency module is introduced to enforce feature consistency across varying viewpoints and enhance spatial understanding. In the second stage, we fine-tune the model on complete scene-level point clouds, which are sparser and structurally more complex. We aggregate the partial vocabularies associated with each scene and generate pseudo labels using the pre-trained model, effectively bridging the semantic gap between dense partial observations and large-scale 3D environments. Extensive experiments on ScanNet, ScanNet200, and S3DIS benchmarks demonstrate that PGOV3D achieves competitive performance in open-vocabulary 3D semantic segmentation.