# ShapeCaptioner: Generative Caption Network for 3D Shapes by Learning a Mapping from Parts Detected in Multiple Views to Sentences

arXiv: https://arxiv.org/abs/1908.00120 · HTML: https://mlsanity.com/paper/1908.00120/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jul 2019.

## Built on by (most cited first, 30 of 30)

- [Scalable 3D Captioning with Pretrained Models](https://mlsanity.com/paper/2306.07279/) (2023)
- [SnowflakeNet: Point Cloud Completion by Snowflake Point Deconvolution with Skip-Transformer](https://mlsanity.com/paper/2108.04444/) (2021)
- [Towards Implicit Text-Guided 3D Shape Generation](https://mlsanity.com/paper/2203.14622/) (2022)
- [Reconstructing Surfaces for Sparse Point Clouds with On-Surface Priors](https://mlsanity.com/paper/2204.10603/) (2022)
- [Surface Reconstruction from Point Clouds by Learning Predictive Context Priors](https://mlsanity.com/paper/2204.11015/) (2022)
- [Towards Better Gradient Consistency for Neural Signed Distance Functions via Level Set Alignment](https://mlsanity.com/paper/2305.11601/) (2023)
- [ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model](https://mlsanity.com/paper/2311.17618/) (2023)
- [Snowflake Point Deconvolution for Point Cloud Completion and Generation with Skip-Transformer](https://mlsanity.com/paper/2202.09367/) (2022)
- [Learning a More Continuous Zero Level Set in Unsigned Distance Fields through Level Set Projection](https://mlsanity.com/paper/2308.11441/) (2023)
- [ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenes](https://mlsanity.com/paper/2212.06250/) (2022)
- [Unsupervised Inference of Signed Distance Functions from Single Sparse Point Clouds without Learning Priors](https://mlsanity.com/paper/2303.14505/) (2023)
- [Learning Signed Distance Functions from Noisy 3D Point Clouds via Noise to Noise Mapping](https://mlsanity.com/paper/2306.01405/) (2023)
- [Unsupervised Learning of Fine Structure Generation for 3D Point Clouds by 2D Projection Matching](https://mlsanity.com/paper/2108.03746/) (2021)
- [Latent Partition Implicit with Surface Codes for 3D Representation](https://mlsanity.com/paper/2207.08631/) (2022)
- [MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Mesh](https://mlsanity.com/paper/2508.01242/) (2025)
- [TriCoLo: Trimodal Contrastive Loss for Text to Shape Retrieval](https://mlsanity.com/paper/2201.07366/) (2022)
- [Parts2Words: Learning Joint Embedding of Point Clouds and Texts by Bidirectional Matching between Parts and Words](https://mlsanity.com/paper/2107.01872/) (2021)
- [T2TD: Text-3D Generation Model based on Prior Knowledge Guidance](https://mlsanity.com/paper/2305.15753/) (2023)
- [GridPull: Towards Scalability in Learning Implicit Representations from 3D Point Clouds](https://mlsanity.com/paper/2308.13175/) (2023)
- [Leveraging VLM-Based Pipelines to Annotate 3D Objects](https://mlsanity.com/paper/2311.17851/) (2023)
- [Inferring Neural Signed Distance Functions by Overfitting on Single Noisy Point Clouds through Finetuning Data-Driven based Priors](https://mlsanity.com/paper/2410.19680/) (2024)
- [Language-Assisted 3D Feature Learning for Semantic Scene Understanding](https://mlsanity.com/paper/2211.14091/) (2022)
- [Fine-grained Text and Image Guided Point Cloud Completion with CLIP Model](https://mlsanity.com/paper/2308.08754/) (2023)
- [Single Image 3D Object Estimation with Primitive Graph Networks](https://mlsanity.com/paper/2109.04153/) (2021)
- [Learning Local Pattern Modularization for Point Cloud Reconstruction from Unseen Classes](https://mlsanity.com/paper/2408.14279/) (2024)
- [Point Cloud Completion Guided by Prior Knowledge via Causal Inference](https://mlsanity.com/paper/2305.17770/) (2023)
- [DFG-PCN: Point Cloud Completion with Degree-Flexible Point Graph](https://mlsanity.com/paper/2509.23703/) (2025)
- [Semantic Validation in Structure from Motion](https://mlsanity.com/paper/2304.02420/) (2023)
- [Diff-3DCap: Shape Captioning with Diffusion Models](https://mlsanity.com/paper/2509.23718/) (2025)
- [Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Control](https://mlsanity.com/paper/2512.24826/) (2025)

## Abstract (arXiv)

3D shape captioning is a challenging application in 3D shape understanding. Captions from recent multi-view based methods reveal that they cannot capture part-level characteristics of 3D shapes. This leads to a lack of detailed part-level description in captions, which human tend to focus on. To resolve this issue, we propose ShapeCaptioner, a generative caption network, to perform 3D shape captioning from semantic parts detected in multiple views. Our novelty lies in learning the knowledge of part detection in multiple views from 3D shape segmentations and transferring this knowledge to facilitate learning the mapping from 3D shapes to sentences. Specifically, ShapeCaptioner aggregates the parts detected in multiple colored views using our novel part class specific aggregation to represent a 3D shape, and then, employs a sequence to sequence model to generate the caption. Our outperforming results show that ShapeCaptioner can learn 3D shape features with more detailed part characteristics to facilitate better 3D shape captioning than previous work.