# RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials

arXiv: https://arxiv.org/abs/1901.01535 · HTML: https://mlsanity.com/paper/1901.01535/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jan 2019.

## Built on by (most cited first, 26 of 26)

- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction](https://mlsanity.com/paper/2206.00665/) (2022)
- [Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks](https://mlsanity.com/paper/2103.10429/) (2021)
- [gradSim: Differentiable simulation for system identification and visuomotor control](https://mlsanity.com/paper/2104.02646/) (2021)
- [Learning Signed Distance Field for Multi-view Surface Reconstruction](https://mlsanity.com/paper/2108.09964/) (2021)
- [Multi-view 3D Reconstruction with Transformer](https://mlsanity.com/paper/2103.12957/) (2021)
- [VolumeFusion: Deep Depth Fusion for 3D Scene Reconstruction](https://mlsanity.com/paper/2108.08623/) (2021)
- [Continual Neural Mapping: Learning An Implicit Scene Representation from Sequential Observations](https://mlsanity.com/paper/2108.05851/) (2021)
- [Ray Priors through Reprojection: Improving Neural Radiance Fields for Novel View Extrapolation](https://mlsanity.com/paper/2205.05922/) (2022)
- [ALTO: Alternating Latent Topologies for Implicit 3D Reconstruction](https://mlsanity.com/paper/2212.04096/) (2022)
- [DebSDF: Delving into the Details and Bias of Neural Indoor Scene Reconstruction](https://mlsanity.com/paper/2308.15536/) (2023)
- [3DRIMR: 3D Reconstruction and Imaging via mmWave Radar based on Deep Learning](https://mlsanity.com/paper/2108.02858/) (2021)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [Deep Surface Reconstruction from Point Clouds with Visibility Information](https://mlsanity.com/paper/2202.01810/) (2022)
- [GARNet: Global-Aware Multi-View 3D Reconstruction Network and the Cost-Performance Tradeoff](https://mlsanity.com/paper/2211.02299/) (2022)
- [R2P: A Deep Learning Model from mmWave Radar to Point Cloud](https://mlsanity.com/paper/2207.10690/) (2022)
- [UMIFormer: Mining the Correlations between Similar Tokens for Multi-View 3D Reconstruction](https://mlsanity.com/paper/2302.13987/) (2023)
- [DeepPoint: A Deep Learning Model for 3D Reconstruction in Point Clouds via mmWave Radar](https://mlsanity.com/paper/2109.09188/) (2021)
- [Long-Range Grouping Transformer for Multi-View 3D Reconstruction](https://mlsanity.com/paper/2308.08724/) (2023)
- [Underground Mapping and Localization Based on Ground-Penetrating Radar](https://mlsanity.com/paper/2409.16446/) (2024)
- [ViiNeuS: Volumetric Initialization for Implicit Neural Surface reconstruction of urban scenes with limited image overlap](https://mlsanity.com/paper/2403.10344/) (2024)
- [Neural Weighted A*: Learning Graph Costs and Heuristics with Differentiable Anytime A*](https://mlsanity.com/paper/2105.01480/) (2021)
- [FoV-Net: Rotation-Invariant CAD B-rep Learning via Field-of-View Ray Casting](https://mlsanity.com/paper/2602.24084/) (2026)
- [UW-SDF: Exploiting Hybrid Geometric Priors for Neural SDF Reconstruction from Underwater Multi-view Monocular Images](https://mlsanity.com/paper/2410.08092/) (2024)
- [View Transformation Robustness for Multi-View 3D Object Reconstruction with Reconstruction Error-Guided View Selection](https://mlsanity.com/paper/2412.11428/) (2024)
- [NERFBK: A High-Quality Benchmark for NERF-Based 3D Reconstruction](https://mlsanity.com/paper/2306.06300/) (2023)

## Abstract (arXiv)

In this paper, we consider the problem of reconstructing a dense 3D model using images captured from different views. Recent methods based on convolutional neural networks (CNN) allow learning the entire task from data. However, they do not incorporate the physics of image formation such as perspective geometry and occlusion. Instead, classical approaches based on Markov Random Fields (MRF) with ray-potentials explicitly model these physical processes, but they cannot cope with large surface appearance variations across different viewpoints. In this paper, we propose RayNet, which combines the strengths of both frameworks. RayNet integrates a CNN that learns view-invariant feature representations with an MRF that explicitly encodes the physics of perspective projection and occlusion. We train RayNet end-to-end using empirical risk minimization. We thoroughly evaluate our approach on challenging real-world datasets and demonstrate its benefits over a piece-wise trained baseline, hand-crafted models as well as other learning-based approaches.