Improving Neural Surface Reconstruction with Feature Priors from Multi-View Image
Not on the map. Why?
Abstract (arXiv)
Recent advancements in Neural Surface Reconstruction (NSR) have significantly improved multi-view reconstruction when coupled with volume rendering. However, relying solely on photometric consistency in image space falls short of addressing complexities posed by real-world data, including occlusions and non-Lambertian surfaces. To tackle these challenges, we propose an investigation into feature-level consistent loss, aiming to harness valuable feature priors from diverse pretext visual tasks and overcome current limitations. It is crucial to note the existing gap in determining the most effective pretext visual task for enhancing NSR. In this study, we comprehensively explore multi-view feature priors from seven pretext visual tasks, comprising thirteen methods. Our main goal is to strengthen NSR training by considering a wide range of possibilities. Additionally, we examine the impact of varying feature resolutions and evaluate both pixel-wise and patch-wise consistent losses, providing insights into effective strategies for improving NSR performance. By incorporating pre-trained representations from MVSFormer and QuadTree, our approach can generate variations of MVS-NeuS and Match-NeuS, respectively. Our results, analyzed on DTU and EPFL datasets, reveal that feature priors from image matching and multi-view stereo outperform other pretext tasks. Moreover, we discover that extending patch-wise photometric consistency to the feature level surpasses the performance of pixel-wise approaches. These findings underscore the effectiveness of these techniques in enhancing NSR outcomes.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 4 Aug 2024.
Builds on
43 of 57 references.
References · 43
- 2023Segment Anythingcited by 8,158
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
- 2022A ConvNet for the 2020scited by 3,545
- 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
- 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
- 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
- 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
- 2021Volume Rendering of Neural Implicit Surfacescited by 990
- 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
- 2016ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Camerascited by 801
- 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
31 more
- 2020Implicit Geometric Regularization for Learning Shapescited by 707
- 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
- 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 643
- 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
- 2022MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstructioncited by 467
- 2023Neuralangelo: High-Fidelity Neural Surface Reconstructioncited by 464
- 2019Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matchingcited by 414
- 2021RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matchingcited by 348
- 2019DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstructioncited by 332
- 2021PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relightingcited by 332
- 2022Geo-Neus: Geometry-Consistent Neural Implicit Surfaces Learning for Multi-view Reconstructioncited by 211
- 2023Iterative Geometry Encoding Volume for Stereo Matchingcited by 188
- 2018DeepV2D: Video to Depth with Differentiable Structure from Motioncited by 185
- 2019DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracingcited by 179
- 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 158
- 2021Improving neural implicit surfaces geometry with patch warpingcited by 136
- 2022QuadTree Attention for Vision Transformerscited by 129
- 2022Neural 3D Scene Reconstruction with the Manhattan-world Assumptioncited by 123
- 2022HF-NeuS: Improved Surface Reconstruction Using High-Frequency Detailscited by 120
- 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
- 2020Visibility-aware Multi-view Stereo Networkcited by 97
- 2017SfM-Net: Learning of Structure and Motion from Videocited by 89
- 2021Learning Signed Distance Field for Multi-view Surface Reconstructioncited by 76
- 2022GO-Surf: Neural Feature Grid Optimization for Fast, High-Fidelity RGB-D Surface Reconstructioncited by 61
- 2022Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepthcited by 58
- 2022MVSFormer: Multi-View Stereo by Learning Robust Image Features and Temperature-based Depthcited by 50
- 2023EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensorscited by 48
- 2019Learning Unsupervised Multi-View Stereopsis via Robust Photometric Consistencycited by 40
- 2023PET-NeuS: Positional Encoding Tri-Planes for Neural Surfacescited by 37
- 2019MVS^2: Deep Unsupervised Multi-view Stereo with Multi-View Symmetrycited by 30
- 2022Recovering Fine Details for Neural Implicit Surface Reconstructioncited by 15
Built on by
Citing papers per year: 2024: 1; 2025: 1; 2026: 1.
3 papers.