mlsanityData to 14 Oct 2026

3D-RETR: End-to-End Single and Multi-View 3D Reconstruction with Transformers

arXiv 2110.08861 ↗First posted 17 Oct 2021cs.CVcited by 15

Not on the map. Why?

Abstract (arXiv)

3D reconstruction aims to reconstruct 3D objects from 2D views. Previous works for 3D reconstruction mainly focus on feature matching between views or using CNNs as backbones. Recently, Transformers have been shown effective in multiple applications of computer vision. However, whether or not Transformers can be used for 3D reconstruction is still unclear. In this paper, we fill this gap by proposing 3D-RETR, which is able to perform end-to-end 3D REconstruction with TRansformers. 3D-RETR first uses a pretrained Transformer to extract visual features from 2D input images. 3D-RETR then uses another Transformer Decoder to obtain the voxel features. A CNN Decoder then takes as input the voxel features to obtain the reconstructed objects. 3D-RETR is capable of 3D reconstruction from a single view or multiple views. Experimental results on two datasets show that 3DRETR reaches state-of-the-art performance on 3D reconstruction. Additional ablation study also demonstrates that 3D-DETR benefits from using Transformers.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 39 of its 56 references that are in the map. First posted on arXiv 17 Oct 2021.

Builds on

39 of 56 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  10. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  11. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  12. 2020Reformer: The Efficient Transformercited by 1,668
27 more
  1. 2020Rethinking Attention with Performerscited by 1,601
  2. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  3. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  4. 2017Focal Loss for Dense Object Detectioncited by 752
  5. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 700
  6. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 651
  7. 2017Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentationscited by 620
  8. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 588
  9. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  10. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  11. 2020Learning Texture Transformer Network for Image Super-Resolutioncited by 347
  12. 2018Pix3D: Dataset and Methods for Single-Image 3D Shape Modelingcited by 218
  13. 2017Learning a Multi-View Stereo Machinecited by 194
  14. 2017Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputscited by 189
  15. 2017Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistencycited by 177
  16. 2019Pix2Vox: Context-aware 3D Reconstruction from Single and Multi-view Imagescited by 164
  17. 2018Scan2CAD: Learning CAD Model Alignment in RGB-D Scanscited by 142
  18. 2015Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Viewscited by 94
  19. 2019Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Eracited by 87
  20. 2020Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Imagescited by 79
  21. 2018Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Predictioncited by 52
  22. 2021Multi-view 3D Reconstruction with Transformercited by 52
  23. 2018Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstructioncited by 44
  24. 20183D-LMNet: Latent Embedding Matching for Accurate and Diverse 3D Point Cloud Reconstruction from a Single Imagecited by 40
  25. 2018Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layerscited by 33
  26. 2017Deep Multitask Architecture for Integrated 2D and 3D Human Sensingcited by 21
  27. 20173D Reconstruction of Incomplete Archaeological Objects Using a Generative Adversarial Networkcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 6; 2024: 4; 2025: 1; 2026: 1.

15 papers.

Sort:

By area:

  1. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
  2. 2023DDF-HO: Hand-Held Object Reconstruction via Conditional Directed Distance Fieldcited by 17
  3. 2022Semi-Supervised Single-View 3D Reconstruction via Prototype Shape Priorscited by 14
  4. 2022GARNet: Global-Aware Multi-View 3D Reconstruction Network and the Cost-Performance Tradeoffcited by 8
  5. 2023UMIFormer: Mining the Correlations between Similar Tokens for Multi-View 3D Reconstructioncited by 7
  6. 2022IC3D: Image-Conditioned 3D Diffusion for Shape Generationcited by 5
  7. 2023Long-Range Grouping Transformer for Multi-View 3D Reconstructioncited by 5
  8. 2024TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assemblycited by 3
  9. 2024M3D: Dual-Stream Selective State Spaces and Depth-Driven Framework for High-Fidelity Single-View 3D Reconstructioncited by 3
  10. 2025Towards General Modality Translation with Contrastive and Predictive Latent Diffusion Bridgecited by 2
  11. 2023Learning to Generate 3D Representations of Building Roofs Using Single-View Aerial Imagerycited by 1
  12. 2023R3D-SWIN:Use Shifted Window Attention for Single-View 3D Reconstructioncited by 1
3 more
  1. 2024View Transformation Robustness for Multi-View 3D Object Reconstruction with Reconstruction Error-Guided View Selectioncited by 1
  2. 2024Shape2.5D: A Dataset of Texture-less Surfaces for Depth and Normals Estimationcited by 0
  3. 2026Structured Diffusion Bridges: Inductive Bias for Denoising Diffusion Bridgescited by 0