# GARNet: Global-Aware Multi-View 3D Reconstruction Network and the Cost-Performance Tradeoff

arXiv: https://arxiv.org/abs/2211.02299 · HTML: https://mlsanity.com/paper/2211.02299/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 46 references that are in the map.
- First posted on arXiv 4 Nov 2022.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [DeepMVS: Learning Multi-view Stereopsis](https://mlsanity.com/paper/1804.00650/) (2018)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [Learning a Multi-View Stereo Machine](https://mlsanity.com/paper/1708.05375/) (2017)
- [Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs](https://mlsanity.com/paper/1703.09438/) (2017)
- [Learning a Predictable and Generative Vector Representation for Objects](https://mlsanity.com/paper/1603.08637/) (2016)
- [BAM: Bottleneck Attention Module](https://mlsanity.com/paper/1807.06514/) (2018)
- [Pix2Vox: Context-aware 3D Reconstruction from Single and Multi-view Images](https://mlsanity.com/paper/1901.11153/) (2019)
- [A Survey of Structure from Motion](https://mlsanity.com/paper/1701.08493/) (2017)
- [Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction](https://mlsanity.com/paper/1706.07036/) (2017)
- [Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation](https://mlsanity.com/paper/1908.01491/) (2019)
- [Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views](https://mlsanity.com/paper/1505.05641/) (2015)
- [Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Images](https://mlsanity.com/paper/2006.12250/) (2020)
- [Improved Adversarial Systems for 3D Object Generation and Reconstruction](https://mlsanity.com/paper/1707.09557/) (2017)
- [Multi-view 3D Reconstruction with Transformer](https://mlsanity.com/paper/2103.12957/) (2021)
- [Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction](https://mlsanity.com/paper/1808.00758/) (2018)
- [Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers](https://mlsanity.com/paper/1804.10975/) (2018)
- [RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials](https://mlsanity.com/paper/1901.01535/) (2019)
- [LegoFormer: Transformers for Block-by-Block Multi-view 3D Reconstruction](https://mlsanity.com/paper/2106.12102/) (2021)
- [3D-RETR: End-to-End Single and Multi-View 3D Reconstruction with Transformers](https://mlsanity.com/paper/2110.08861/) (2021)

## Built on by (most cited first, 8 of 8)

- [Efficient Large-scale Scene Representation with a Hybrid of High-resolution Grid and Plane Features](https://mlsanity.com/paper/2303.03003/) (2023)
- [UMIFormer: Mining the Correlations between Similar Tokens for Multi-View 3D Reconstruction](https://mlsanity.com/paper/2302.13987/) (2023)
- [Long-Range Grouping Transformer for Multi-View 3D Reconstruction](https://mlsanity.com/paper/2308.08724/) (2023)
- [UltraTwin: Towards Cardiac Anatomical Twin Generation from Multi-view 2D Ultrasound](https://mlsanity.com/paper/2506.23490/) (2025)
- [A survey on deep learning approaches for data integration in autonomous driving system](https://mlsanity.com/paper/2306.11740/) (2023)
- [R3D-SWIN:Use Shifted Window Attention for Single-View 3D Reconstruction](https://mlsanity.com/paper/2312.02725/) (2023)
- [View Transformation Robustness for Multi-View 3D Object Reconstruction with Reconstruction Error-Guided View Selection](https://mlsanity.com/paper/2412.11428/) (2024)
- [PointCHR: Point Cloud Analysis via Curvature-Aware Hyperbolic Rectification](https://mlsanity.com/paper/2607.24052/) (2026)

## Abstract (arXiv)

Deep learning technology has made great progress in multi-view 3D reconstruction tasks. At present, most mainstream solutions establish the mapping between views and shape of an object by assembling the networks of 2D encoder and 3D decoder as the basic structure while they adopt different approaches to obtain aggregation of features from several views. Among them, the methods using attention-based fusion perform better and more stable than the others, however, they still have an obvious shortcoming -- the strong independence of each view during predicting the weights for merging leads to a lack of adaption of the global state. In this paper, we propose a global-aware attention-based fusion approach that builds the correlation between each branch and the global to provide a comprehensive foundation for weights inference. In order to enhance the ability of the network, we introduce a novel loss function to supervise the shape overall and propose a dynamic two-stage training strategy that can effectively adapt to all reconstructors with attention-based fusion. Experiments on ShapeNet verify that our method outperforms existing SOTA methods while the amount of parameters is far less than the same type of algorithm, Pix2Vox++. Furthermore, we propose a view-reduction method based on maximizing diversity and discuss the cost-performance tradeoff of our model to achieve a better performance when facing heavy input amount and limited computational cost.