# PartGlot: Learning Shape Part Segmentation from Language Reference Games

arXiv: https://arxiv.org/abs/2112.06390 · HTML: https://mlsanity.com/paper/2112.06390/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 49 references that are in the map.
- First posted on arXiv 13 Dec 2021.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers](https://mlsanity.com/paper/2108.08839/) (2021)
- [CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation](https://mlsanity.com/paper/2110.02624/) (2021)
- [Learning Shape Templates with Structured Implicit Functions](https://mlsanity.com/paper/1904.06447/) (2019)
- [BSP-Net: Generating Compact Meshes via Binary Space Partitioning](https://mlsanity.com/paper/1911.06971/) (2019)
- [Learning Shape Abstractions by Assembling Volumetric Primitives](https://mlsanity.com/paper/1612.00404/) (2016)
- [CvxNet: Learnable Convex Decomposition](https://mlsanity.com/paper/1909.05736/) (2019)
- [InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referring](https://mlsanity.com/paper/2103.01128/) (2021)
- [Superquadrics Revisited: Learning 3D Shape Parsing beyond Cuboids](https://mlsanity.com/paper/1904.09970/) (2019)
- [How to represent part-whole hierarchies in a neural network](https://mlsanity.com/paper/2102.12627/) (2021)
- [Generative Adversarial Transformers](https://mlsanity.com/paper/2103.01209/) (2021)
- [Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks](https://mlsanity.com/paper/2103.10429/) (2021)
- [BAE-NET: Branched Autoencoder for Shape Co-Segmentation](https://mlsanity.com/paper/1903.11228/) (2019)
- [LanguageRefer: Spatial-Language Model for 3D Visual Grounding](https://mlsanity.com/paper/2107.03438/) (2021)
- [ShapeGlot: Learning Language for Shape Differentiation](https://mlsanity.com/paper/1905.02925/) (2019)
- [Language Grounding with 3D Objects](https://mlsanity.com/paper/2107.12514/) (2021)

## Built on by (most cited first, 30 of 30)

- [Multi3DRefer: Grounding Text Description to Multiple 3D Objects](https://mlsanity.com/paper/2309.05251/) (2023)
- [State of the Art on Diffusion Models for Visual Computing](https://mlsanity.com/paper/2310.07204/) (2023)
- [PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Models](https://mlsanity.com/paper/2212.01558/) (2022)
- [SALAD: Part-Level Latent Diffusion for 3D Shape Generation and Manipulation](https://mlsanity.com/paper/2303.12236/) (2023)
- [SATR: Zero-Shot Semantic Segmentation of 3D Shapes](https://mlsanity.com/paper/2304.04909/) (2023)
- [Segment Any Mesh](https://mlsanity.com/paper/2408.13679/) (2024)
- [ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenes](https://mlsanity.com/paper/2212.06250/) (2022)
- [DiffFacto: Controllable Part-Based 3D Point Cloud Generation with Cross Diffusion](https://mlsanity.com/paper/2305.01921/) (2023)
- [Mesh2Tex: Generating Mesh Textures from Image Queries](https://mlsanity.com/paper/2304.05868/) (2023)
- [MeshSegmenter: Zero-Shot Mesh Semantic Segmentation via Texture Synthesis](https://mlsanity.com/paper/2407.13675/) (2024)
- [Text2Scene: Text-driven Indoor Scene Stylization with Part-aware Details](https://mlsanity.com/paper/2308.16880/) (2023)
- [ZeroPS: High-quality Cross-modal Knowledge Transfer for Zero-Shot 3D Part Segmentation](https://mlsanity.com/paper/2311.14262/) (2023)
- [PartSTAD: 2D-to-3D Part Segmentation Task Adaptation](https://mlsanity.com/paper/2401.05906/) (2024)
- [Investigating Agency of LLMs in Human-AI Collaboration Tasks](https://mlsanity.com/paper/2305.12815/) (2023)
- [3D Part Segmentation via Geometric Aggregation of 2D Visual Features](https://mlsanity.com/paper/2412.04247/) (2024)
- [Iterative Superquadric Recomposition of 3D Objects from Multiple Views](https://mlsanity.com/paper/2309.02102/) (2023)
- [Voxel-informed Language Grounding](https://mlsanity.com/paper/2205.09710/) (2022)
- [LADIS: Language Disentanglement for 3D Shape Editing](https://mlsanity.com/paper/2212.05011/) (2022)
- [Leveraging VLM-Based Pipelines to Annotate 3D Objects](https://mlsanity.com/paper/2311.17851/) (2023)
- [Advances in 3D Neural Stylization: A Survey](https://mlsanity.com/paper/2311.18328/) (2023)
- [DAE-Net: Deforming Auto-Encoder for fine-grained shape co-segmentation](https://mlsanity.com/paper/2311.13125/) (2023)
- [Language-Assisted 3D Feature Learning for Semantic Scene Understanding](https://mlsanity.com/paper/2211.14091/) (2022)
- [SPIN: Hierarchical Segmentation with Subpart Granularity in Natural Images](https://mlsanity.com/paper/2407.09686/) (2024)
- [Gesturing Toward Abstraction: Multimodal Convention Formation in Collaborative Physical Tasks](https://mlsanity.com/paper/2602.08914/) (2026)
- [Which One? Leveraging Context Between Objects and Multiple Views for Language Grounding](https://mlsanity.com/paper/2311.06694/) (2023)
- [Advancements in Point Cloud-Based 3D Defect Detection and Classification for Industrial Systems: A Comprehensive Survey](https://mlsanity.com/paper/2402.12923/) (2024)
- [Name That Part: 3D Part Segmentation and Naming](https://mlsanity.com/paper/2512.18003/) (2025)
- [Deep Sketch-Based 3D Modeling: A Survey](https://mlsanity.com/paper/2603.03287/) (2026)
- [PinPoint3D: Fine-Grained 3D Part Segmentation from a Few Clicks](https://mlsanity.com/paper/2509.25970/) (2025)
- [Hierarchical Transformers for Unsupervised 3D Shape Abstraction](https://mlsanity.com/paper/2510.27088/) (2025)

## Abstract (arXiv)

We introduce PartGlot, a neural framework and associated architectures for learning semantic part segmentation of 3D shape geometry, based solely on part referential language. We exploit the fact that linguistic descriptions of a shape can provide priors on the shape's parts -- as natural language has evolved to reflect human perception of the compositional structure of objects, essential to their recognition and use. For training, we use the paired geometry / language data collected in the ShapeGlot work for their reference game, where a speaker creates an utterance to differentiate a target shape from two distractors and the listener has to find the target based on this utterance. Our network is designed to solve this target discrimination problem, carefully incorporating a Transformer-based attention module so that the output attention can precisely highlight the semantic part or parts described in the language. Furthermore, the network operates without any direct supervision on the 3D geometry itself. Surprisingly, we further demonstrate that the learned part information is generalizable to shape classes unseen during training. Our approach opens the possibility of learning 3D shape parts from language alone, without the need for large-scale part geometry annotations, thus facilitating annotation acquisition.