# LegoFormer: Transformers for Block-by-Block Multi-view 3D Reconstruction

arXiv: https://arxiv.org/abs/2106.12102 · HTML: https://mlsanity.com/paper/2106.12102/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 50 references that are in the map.
- First posted on arXiv 23 Jun 2021.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generation](https://mlsanity.com/paper/1802.05384/) (2018)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion](https://mlsanity.com/paper/2003.01456/) (2020)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [Understanding the Difficulty of Training Transformers](https://mlsanity.com/paper/2004.08249/) (2020)
- [Learning a Multi-View Stereo Machine](https://mlsanity.com/paper/1708.05375/) (2017)
- [Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs](https://mlsanity.com/paper/1703.09438/) (2017)
- [Pix2Vox: Context-aware 3D Reconstruction from Single and Multi-view Images](https://mlsanity.com/paper/1901.11153/) (2019)
- [A Survey of Structure from Motion](https://mlsanity.com/paper/1701.08493/) (2017)
- [Transformers without Tears: Improving the Normalization of Self-Attention](https://mlsanity.com/paper/1910.05895/) (2019)
- [Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Images](https://mlsanity.com/paper/2006.12250/) (2020)
- [Multi-view 3D Reconstruction with Transformer](https://mlsanity.com/paper/2103.12957/) (2021)
- [Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction](https://mlsanity.com/paper/1808.00758/) (2018)
- [Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers](https://mlsanity.com/paper/1804.10975/) (2018)
- [Recent Developments in Boolean Matrix Factorization](https://mlsanity.com/paper/2012.03127/) (2020)

## Built on by (most cited first, 25 of 25)

- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [RealFusion: 360deg Reconstruction of Any Object from a Single Image](https://mlsanity.com/paper/2302.10663/) (2023)
- [PC2: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstruction](https://mlsanity.com/paper/2302.10668/) (2023)
- [AGG: Amortized Generative 3D Gaussians for Single Image to 3D](https://mlsanity.com/paper/2401.04099/) (2024)
- [Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI](https://mlsanity.com/paper/2311.03783/) (2023)
- [Voxel-informed Language Grounding](https://mlsanity.com/paper/2205.09710/) (2022)
- [Delving Deep into Pixel Alignment Feature for Accurate Multi-view Human Mesh Recovery](https://mlsanity.com/paper/2301.06020/) (2023)
- [CCD-3DR: Consistent Conditioning in Diffusion for Single-Image 3D Reconstruction](https://mlsanity.com/paper/2308.07837/) (2023)
- [GARNet: Global-Aware Multi-View 3D Reconstruction Network and the Cost-Performance Tradeoff](https://mlsanity.com/paper/2211.02299/) (2022)
- [RGB2Point: 3D Point Cloud Generation from Single RGB Images](https://mlsanity.com/paper/2407.14979/) (2024)
- [UMIFormer: Mining the Correlations between Similar Tokens for Multi-View 3D Reconstruction](https://mlsanity.com/paper/2302.13987/) (2023)
- [3D-C2FT: Coarse-to-fine Transformer for Multi-view 3D Reconstruction](https://mlsanity.com/paper/2205.14575/) (2022)
- [Long-Range Grouping Transformer for Multi-View 3D Reconstruction](https://mlsanity.com/paper/2308.08724/) (2023)
- [ConRad: Image Constrained Radiance Fields for 3D Generation from a Single Image](https://mlsanity.com/paper/2311.05230/) (2023)
- [6Img-to-3D: Few-Image Large-Scale Outdoor Driving Scene Reconstruction](https://mlsanity.com/paper/2404.12378/) (2024)
- [Which One? Leveraging Context Between Objects and Multiple Views for Language Grounding](https://mlsanity.com/paper/2311.06694/) (2023)
- [Multi-Task Domain Adaptation for Language Grounding with 3D Objects](https://mlsanity.com/paper/2407.02846/) (2024)
- [Automatic Reverse Engineering: Creating computer-aided design (CAD) models from multi-view images](https://mlsanity.com/paper/2309.13281/) (2023)
- [R3D-SWIN:Use Shifted Window Attention for Single-View 3D Reconstruction](https://mlsanity.com/paper/2312.02725/) (2023)
- [DiffPoint: Single and Multi-view Point Cloud Reconstruction with ViT Based Diffusion Model](https://mlsanity.com/paper/2402.11241/) (2024)
- [View Transformation Robustness for Multi-View 3D Object Reconstruction with Reconstruction Error-Guided View Selection](https://mlsanity.com/paper/2412.11428/) (2024)
- [3D-COCO: extension of MS-COCO dataset for image detection and 3D reconstruction modules](https://mlsanity.com/paper/2404.05641/) (2024)
- [Multispectral-NeRF:a multispectral modeling approach based on neural radiance fields](https://mlsanity.com/paper/2509.11169/) (2025)
- [QAL: A Loss for Recall Precision Balance in 3D Reconstruction](https://mlsanity.com/paper/2511.17824/) (2025)
- [Point-MF: Stabilizing One-Step Mean Flows for Single-Image Point Cloud Reconstruction](https://mlsanity.com/paper/2604.24586/) (2026)

## Abstract (arXiv)

Most modern deep learning-based multi-view 3D reconstruction techniques use RNNs or fusion modules to combine information from multiple images after independently encoding them. These two separate steps have loose connections and do not allow easy information sharing among views. We propose LegoFormer, a transformer model for voxel-based 3D reconstruction that uses the attention layers to share information among views during all computational stages. Moreover, instead of predicting each voxel independently, we propose to parametrize the output with a series of low-rank decomposition factors. This reformulation allows the prediction of an object as a set of independent regular structures then aggregated to obtain the final reconstruction. Experiments conducted on ShapeNet demonstrate the competitive performance of our model with respect to the state of the art while having increased interpretability thanks to the self-attention layers. We also show promising generalization results to real data.