VMNet: Voxel-Mesh Network for Geodesic-Aware 3D Semantic Segmentation
Not on the map. Why?
Abstract (arXiv)
In recent years, sparse voxel-based methods have become the state-of-the-arts for 3D semantic segmentation of indoor scenes, thanks to the powerful 3D CNNs. Nevertheless, being oblivious to the underlying geometry, voxel-based methods suffer from ambiguous features on spatially close objects and struggle with handling complex and irregular geometries due to the lack of geodesic information. In view of this, we present Voxel-Mesh Network (VMNet), a novel 3D deep architecture that operates on the voxel and mesh representations leveraging both the Euclidean and geodesic information. Intuitively, the Euclidean information extracted from voxels can offer contextual cues representing interactions between nearby objects, while the geodesic information extracted from meshes can help separate objects that are spatially close but have disconnected surfaces. To incorporate such information from the two domains, we design an intra-domain attentive module for effective feature aggregation and an inter-domain attentive module for adaptive feature fusion. Experimental results validate the effectiveness of VMNet: specifically, on the challenging ScanNet dataset for large-scale segmentation of indoor scenes, it outperforms the state-of-the-art SparseConvNet and MinkowskiNet (74.6% vs 72.5% and 73.6% in mIoU) with a simpler network structure (17M vs 30M and 38M parameters). Code release: https://github.com/hzykent/VMNet
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 55 of its 79 references that are in the map. First posted on arXiv 29 Jul 2021.
Builds on
55 of 79 references.
References · 55
- 2017Attention Is All You Needcited by 59,932
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
- 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
- 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
- 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
- 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
- 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
- 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,085
- 2020Point Transformercited by 1,462
38 more
- 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
- 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
- 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
- 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
- 2018Image Transformercited by 770
- 2018PointCNN: Convolution On X-Transformed Pointscited by 764
- 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
- 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
- 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
- 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 500
- 2016Geometric deep learning on graphs and manifolds using mixture model CNNscited by 468
- 2016Semantic Scene Completion from a Single Depth Imagecited by 468
- 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 422
- 2019Point-Voxel CNN for Efficient 3D Deep Learningcited by 343
- 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 340
- 2017Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphscited by 307
- 2016Volumetric and Multi-View CNNs for Object Classification on 3D Datacited by 240
- 2020PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Samplingcited by 238
- 2020OccuSeg: Occupancy-aware 3D Instance Segmentationcited by 177
- 2018Tangent Convolutions for Dense Prediction in 3Dcited by 173
- 2018SPLATNet: Sparse Lattice Networks for Point Cloud Processingcited by 158
- 2017Pointwise Convolutional Neural Networkscited by 126
- 20183DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentationcited by 118
- 2017ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scanscited by 103
- 2020Virtual Multi-view Fusion for 3D Semantic Segmentationcited by 99
- 2016Learning shape correspondence with anisotropic convolutional neural networkscited by 95
- 2021Bidirectional Projection Network for Cross Dimension Scene Understandingcited by 95
- 2019Interpolated Convolutional Networks for 3D Point Cloud Understandingcited by 86
- 2017Deep Projective 3D Semantic Segmentationcited by 81
- 2019Multi-view PointNet for 3D Scene Understandingcited by 76
- 2019Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentationcited by 68
- 2019A-CNN: Annularly Convolutional Neural Networks on Point Cloudscited by 66
- 2018Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Cloudscited by 62
- 2020JSENet: Joint Semantic Segmentation and Edge Detection Network for 3D Point Cloudscited by 62
- 2019Spherical Kernel for Efficient Graph Convolution on 3D Point Cloudscited by 57
- 2019Octree guided CNN with Spherical Kernels for 3D Point Cloudscited by 50
- 2018TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshescited by 45
- 2018Fully-Convolutional Point Networks for Large-Scale Point Cloudscited by 44
50 most cited of 55 All 55 (JSON)
Built on by
Citing papers per year: 2022: 1; 2023: 1.
2 papers.