# Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction

arXiv: https://arxiv.org/abs/1808.00758 · HTML: https://mlsanity.com/paper/1808.00758/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2018.

## Built on by (most cited first, 30 of 44)

- [Multi-view 3D Reconstruction with Transformer](https://mlsanity.com/paper/2103.12957/) (2021)
- [Learning Semantic Segmentation of Large-Scale Point Clouds with Random Sampling](https://mlsanity.com/paper/2107.02389/) (2021)
- [You Never Cluster Alone](https://mlsanity.com/paper/2106.01908/) (2021)
- [VoRTX: Volumetric 3D Reconstruction With Transformers for Voxelwise View Selection and Fusion](https://mlsanity.com/paper/2112.00236/) (2021)
- [NVFi: Neural Velocity Fields for 3D Physics Learning from Dynamic Videos](https://mlsanity.com/paper/2312.06398/) (2023)
- [LegoFormer: Transformers for Block-by-Block Multi-view 3D Reconstruction](https://mlsanity.com/paper/2106.12102/) (2021)
- [RangeUDF: Semantic Surface Reconstruction from 3D Point Clouds](https://mlsanity.com/paper/2204.09138/) (2022)
- [Quark: Real-time, High-resolution, and General Neural View Synthesis](https://mlsanity.com/paper/2411.16680/) (2024)
- [3D-RETR: End-to-End Single and Multi-View 3D Reconstruction with Transformers](https://mlsanity.com/paper/2110.08861/) (2021)
- [Gaussian Radar Transformer for Semantic Segmentation in Noisy Radar Data](https://mlsanity.com/paper/2212.03690/) (2022)
- [RMS-FlowNet: Efficient and Robust Multi-Scale Scene Flow Estimation for Large-Scale Point Clouds](https://mlsanity.com/paper/2204.00354/) (2022)
- [Tree-D Fusion: Simulation-Ready Tree Dataset from Single Images with Diffusion Priors](https://mlsanity.com/paper/2407.10330/) (2024)
- [RayDF: Neural Ray-surface Distance Fields with Multi-view Consistency](https://mlsanity.com/paper/2310.19629/) (2023)
- [Data-Driven 3D Reconstruction of Dressed Humans From Sparse Views](https://mlsanity.com/paper/2104.08013/) (2021)
- [GARNet: Global-Aware Multi-View 3D Reconstruction Network and the Cost-Performance Tradeoff](https://mlsanity.com/paper/2211.02299/) (2022)
- [DLA-Net: Learning Dual Local Attention Features for Semantic Segmentation of Large-Scale Building Facade Point Clouds](https://mlsanity.com/paper/2106.00376/) (2021)
- [PointVST: Self-Supervised Pre-training for 3D Point Clouds via View-Specific Point-to-Image Translation](https://mlsanity.com/paper/2212.14197/) (2022)
- [UMIFormer: Mining the Correlations between Similar Tokens for Multi-View 3D Reconstruction](https://mlsanity.com/paper/2302.13987/) (2023)
- [Dense Voxel 3D Reconstruction Using a Monocular Event Camera](https://mlsanity.com/paper/2309.00385/) (2023)
- [Set-to-Sequence Methods in Machine Learning: a Review](https://mlsanity.com/paper/2103.09656/) (2021)
- [SoftPool++: An Encoder-Decoder Network for Point Cloud Completion](https://mlsanity.com/paper/2205.03899/) (2022)
- [3D-C2FT: Coarse-to-fine Transformer for Multi-view 3D Reconstruction](https://mlsanity.com/paper/2205.14575/) (2022)
- [Long-Range Grouping Transformer for Multi-View 3D Reconstruction](https://mlsanity.com/paper/2308.08724/) (2023)
- [Similarity-Aware Fusion Network for 3D Semantic Segmentation](https://mlsanity.com/paper/2107.01579/) (2021)
- [PICASO: Permutation-Invariant Cascaded Attentional Set Operator](https://mlsanity.com/paper/2107.08305/) (2021)
- [VPFusion: Joint 3D Volume and Pixel-Aligned Feature Fusion for Single and Multi-view 3D Reconstruction](https://mlsanity.com/paper/2203.07553/) (2022)
- [Set Interdependence Transformer: Set-to-Sequence Neural Networks for Permutation Learning and Structure Prediction](https://mlsanity.com/paper/2206.03720/) (2022)
- [Radar Velocity Transformer: Single-scan Moving Object Segmentation in Noisy Radar Point Clouds](https://mlsanity.com/paper/2507.03463/) (2025)
- [DiffPoint: Single and Multi-view Point Cloud Reconstruction with ViT Based Diffusion Model](https://mlsanity.com/paper/2402.11241/) (2024)
- [RMS-FlowNet++: Efficient and Robust Multi-Scale Scene Flow Estimation for Large-Scale Point Clouds](https://mlsanity.com/paper/2407.01129/) (2024)

## Abstract (arXiv)

We study the problem of recovering an underlying 3D shape from a set of images. Existing learning based approaches usually resort to recurrent neural nets, e.g., GRU, or intuitive pooling operations, e.g., max/mean poolings, to fuse multiple deep features encoded from input images. However, GRU based approaches are unable to consistently estimate 3D shapes given different permutations of the same set of input images as the recurrent unit is permutation variant. It is also unlikely to refine the 3D shape given more images due to the long-term memory loss of GRU. Commonly used pooling approaches are limited to capturing partial information, e.g., max/mean values, ignoring other valuable features. In this paper, we present a new feed-forward neural module, named AttSets, together with a dedicated training algorithm, named FASet, to attentively aggregate an arbitrarily sized deep feature set for multi-view 3D reconstruction. The AttSets module is permutation invariant, computationally efficient and flexible to implement, while the FASet algorithm enables the AttSets based network to be remarkably robust and generalize to an arbitrary number of input images. We thoroughly evaluate FASet and the properties of AttSets on multiple large public datasets. Extensive experiments show that AttSets together with FASet algorithm significantly outperforms existing aggregation approaches.