# PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillation

arXiv: https://arxiv.org/abs/2312.04016 · HTML: https://mlsanity.com/paper/2312.04016/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 42 of its 59 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 42)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies](https://mlsanity.com/paper/2206.04670/) (2022)
- [Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs](https://mlsanity.com/paper/1711.09869/) (2017)
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021)
- [GLIPv2: Unifying Localization and Vision-Language Understanding](https://mlsanity.com/paper/2206.05836/) (2022)
- [Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training](https://mlsanity.com/paper/2205.14401/) (2022)
- [Self-Supervised Pretraining of 3D Features on any Point-Cloud](https://mlsanity.com/paper/2101.02691/) (2021)
- [Shape As Points: A Differentiable Poisson Solver](https://mlsanity.com/paper/2106.03452/) (2021)
- [CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP](https://mlsanity.com/paper/2301.04926/) (2023)
- [Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Data](https://mlsanity.com/paper/2203.16258/) (2022)
- [Segment Any Point Cloud Sequences by Distilling Vision Foundation Models](https://mlsanity.com/paper/2306.09347/) (2023)
- [PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Models](https://mlsanity.com/paper/2212.01558/) (2022)
- [DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation](https://mlsanity.com/paper/2307.01831/) (2023)
- [Few-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2006.12052/) (2020)

## Built on by (most cited first, 28 of 28)

- [SAMPart3D: Segment Any Part in 3D Objects](https://mlsanity.com/paper/2411.07184/) (2024)
- [PARTFIELD: Learning 3D Feature Fields for Part Segmentation and Beyond](https://mlsanity.com/paper/2504.11451/) (2025)
- [HoloPart: Generative 3D Part Amodal Segmentation](https://mlsanity.com/paper/2504.07943/) (2025)
- [X-Part: high fidelity and structure coherent shape decomposition](https://mlsanity.com/paper/2509.08643/) (2025)
- [Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modeling](https://mlsanity.com/paper/2502.02590/) (2025)
- [P3-SAM: Native 3D Part Segmentation](https://mlsanity.com/paper/2509.06784/) (2025)
- [Find Any Part in 3D](https://mlsanity.com/paper/2411.13550/) (2024)
- [MeshCoder: LLM-Powered Structured Mesh Code Generation from Point Clouds](https://mlsanity.com/paper/2508.14879/) (2025)
- [3D Part Segmentation via Geometric Aggregation of 2D Visual Features](https://mlsanity.com/paper/2412.04247/) (2024)
- [GeoSAM2: Unleashing the Power of SAM2 for 3D Part Segmentation](https://mlsanity.com/paper/2508.14036/) (2025)
- [SegviGen: Repurposing 3D Generative Model for Part Segmentation](https://mlsanity.com/paper/2603.16869/) (2026)
- [SPARK: Sim-ready Part-level Articulated Reconstruction with VLM Knowledge](https://mlsanity.com/paper/2512.01629/) (2025)
- [PatchAlign3D: Local Feature Alignment for Dense 3D Shape Understanding](https://mlsanity.com/paper/2601.02457/) (2026)
- [Open-Vocabulary Semantic Part Segmentation of 3D Human](https://mlsanity.com/paper/2502.19782/) (2025)
- [MonoArt: Progressive Structural Reasoning for Monocular Articulated 3D Reconstruction](https://mlsanity.com/paper/2603.19231/) (2026)
- [Name That Part: 3D Part Segmentation and Naming](https://mlsanity.com/paper/2512.18003/) (2025)
- [Segment Any 3D-Part in a Scene from a Sentence](https://mlsanity.com/paper/2506.19331/) (2025)
- [Parameter-Efficient CLIP Adaptation for 3D Understanding via Unified Tokenization](https://mlsanity.com/paper/2505.18819/) (2025)
- [EI-Part: Explode for Completion and Implode for Refinement](https://mlsanity.com/paper/2603.14021/) (2026)
- [PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understanding](https://mlsanity.com/paper/2606.06485/) (2026)
- [MR-COSMO: Visual-Text Memory Recall and Direct CrOSs-MOdal Alignment Method for Query-Driven 3D Segmentation](https://mlsanity.com/paper/2506.20991/) (2025)
- [Adaptive Point-Prompt Tuning: Fine-Tuning Heterogeneous Foundation Models for 3D Point Cloud Analysis](https://mlsanity.com/paper/2509.00374/) (2025)
- [SegGraph: Leveraging Graphs of SAM Segments for Few-Shot 3D Part Segmentation](https://mlsanity.com/paper/2512.16143/) (2025)
- [CoSMo3D: Open-World Promptable 3D Semantic Part Segmentation through LLM-Guided Canonical Spatial Modeling](https://mlsanity.com/paper/2603.01205/) (2026)
- [Material Magic Wand: Material-Aware Grouping of 3D Parts in Untextured Meshes](https://mlsanity.com/paper/2603.17370/) (2026)
- [Bridging the Dimensionality Gap: A Taxonomy and Survey of 2D Vision Model Adaptation for 3D Analysis](https://mlsanity.com/paper/2604.03334/) (2026)
- [PartMat: Material-Aware 3D Part Decomposition with a Single Global Latent](https://mlsanity.com/paper/2608.01825/) (2026)
- [Organize Primitives into Semantic Parts: Reinforcement Reasoning for 3D Segmentation](https://mlsanity.com/paper/2610.04602/) (2026)

## Abstract (arXiv)

This paper proposes a cross-modal distillation framework, PartDistill, which transfers 2D knowledge from vision-language models (VLMs) to facilitate 3D shape part segmentation. PartDistill addresses three major challenges in this task: the lack of 3D segmentation in invisible or undetected regions in the 2D projections, inconsistent 2D predictions by VLMs, and the lack of knowledge accumulation across different 3D shapes. PartDistill consists of a teacher network that uses a VLM to make 2D predictions and a student network that learns from the 2D predictions while extracting geometrical features from multiple 3D shapes to carry out 3D part segmentation. A bi-directional distillation, including forward and backward distillations, is carried out within the framework, where the former forward distills the 2D predictions to the student network, and the latter improves the quality of the 2D predictions, which subsequently enhances the final 3D segmentation. Moreover, PartDistill can exploit generative models that facilitate effortless 3D shape creation for generating knowledge sources to be distilled. Through extensive experiments, PartDistill boosts the existing methods with substantial margins on widely used ShapeNetPart and PartNetE datasets, by more than 15% and 12% higher mIoU scores, respectively. The code for this work is available at https://github.com/ardianumam/PartDistill.