# ShapeFormer: Transformer-based Shape Completion via Sparse Representation

arXiv: https://arxiv.org/abs/2201.10326 · HTML: https://mlsanity.com/paper/2201.10326/

## Key facts

- Cited by 106 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 79 references that are in the map.
- First posted on arXiv 25 Jan 2022.

## Builds on (most cited first, 30 of 59)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [PCN: Point Completion Network](https://mlsanity.com/paper/1808.00671/) (2018)
- [Local Implicit Grid Representations for 3D Scenes](https://mlsanity.com/paper/2003.08981/) (2020)

## Built on by (most cited first, 30 of 106)

- [3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models](https://mlsanity.com/paper/2301.11445/) (2023)
- [VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.12251/) (2023)
- [LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes](https://mlsanity.com/paper/2311.13384/) (2023)
- [Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation](https://mlsanity.com/paper/2306.17115/) (2023)
- [AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation](https://mlsanity.com/paper/2203.09516/) (2022)
- [DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior](https://mlsanity.com/paper/2310.16818/) (2023)
- [Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions](https://mlsanity.com/paper/2211.13757/) (2022)
- [Neural Wavelet-domain Diffusion for 3D Shape Generation](https://mlsanity.com/paper/2209.08725/) (2022)
- [Locally Attentional SDF Diffusion for Controllable 3D Shape Generation](https://mlsanity.com/paper/2305.04461/) (2023)
- [Hunyuan3D 2.5: Towards High-Fidelity 3D Assets Generation with Ultimate Details](https://mlsanity.com/paper/2506.16504/) (2025)
- [3DILG: Irregular Latent Grids for 3D Generative Modeling](https://mlsanity.com/paper/2205.13914/) (2022)
- [Multiview Compressive Coding for 3D Reconstruction](https://mlsanity.com/paper/2301.08247/) (2023)
- [SALAD: Part-Level Latent Diffusion for 3D Shape Generation and Manipulation](https://mlsanity.com/paper/2303.12236/) (2023)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model](https://mlsanity.com/paper/2207.09446/) (2022)
- [CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Language](https://mlsanity.com/paper/2211.01427/) (2022)
- [LATTICE: Democratize High-Fidelity 3D Generation at Scale](https://mlsanity.com/paper/2512.03052/) (2025)
- [Surface Reconstruction from Point Clouds: A Survey and a Benchmark](https://mlsanity.com/paper/2205.02413/) (2022)
- [Autoregressive 3D Shape Generation via Canonical Mapping](https://mlsanity.com/paper/2204.01955/) (2022)
- [Point-Cloud Completion with Pretrained Text-to-image Diffusion Models](https://mlsanity.com/paper/2306.10533/) (2023)
- [DiffComplete: Diffusion-based Generative 3D Shape Completion](https://mlsanity.com/paper/2306.16329/) (2023)
- [An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion](https://mlsanity.com/paper/2408.03178/) (2024)
- [MedShapeNet -- A Large-Scale Dataset of 3D Medical Shapes for Computer Vision](https://mlsanity.com/paper/2308.16139/) (2023)
- [Transformers in 3D Point Clouds: A Survey](https://mlsanity.com/paper/2205.07417/) (2022)
- [Neural Wavelet-domain Diffusion for 3D Shape Generation, Inversion, and Manipulation](https://mlsanity.com/paper/2302.00190/) (2023)
- [3DQD: Generalized Deep 3D Shape Prior via Part-Discretized Diffusion Process](https://mlsanity.com/paper/2303.10406/) (2023)
- [GINA-3D: Learning to Generate Implicit Neural Assets in the Wild](https://mlsanity.com/paper/2304.02163/) (2023)
- [Make-A-Shape: a Ten-Million-scale 3D Shape Model](https://mlsanity.com/paper/2401.11067/) (2024)
- [SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation](https://mlsanity.com/paper/2412.01801/) (2024)
- [PatchComplete: Learning Multi-Resolution Patch Priors for 3D Shape Completion on Unseen Categories](https://mlsanity.com/paper/2206.04916/) (2022)

## Abstract (arXiv)

We present ShapeFormer, a transformer-based network that produces a distribution of object completions, conditioned on incomplete, and possibly noisy, point clouds. The resultant distribution can then be sampled to generate likely completions, each exhibiting plausible shape details while being faithful to the input. To facilitate the use of transformers for 3D, we introduce a compact 3D representation, vector quantized deep implicit function, that utilizes spatial sparsity to represent a close approximation of a 3D shape by a short sequence of discrete variables. Experiments demonstrate that ShapeFormer outperforms prior art for shape completion from ambiguous partial inputs in terms of both completion quality and diversity. We also show that our approach effectively handles a variety of shape types, incomplete patterns, and real-world scans.