# Semantic 3D Reconstruction with Continuous Regularization and Ray Potentials Using a Visibility Consistency Constraint

arXiv: https://arxiv.org/abs/1604.02885 · HTML: https://mlsanity.com/paper/1604.02885/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2016.

## Built on by (most cited first, 5 of 5)

- [Neural RGB-D Surface Reconstruction](https://mlsanity.com/paper/2104.04532/) (2021)
- [SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphs](https://mlsanity.com/paper/2404.00469/) (2024)
- [SSDNeRF: Semantic Soft Decomposition of Neural Radiance Fields](https://mlsanity.com/paper/2212.03406/) (2022)
- [Self-Supervised Monocular Scene Decomposition and Depth Estimation](https://mlsanity.com/paper/2110.11275/) (2021)
- [Ab Initio Particle-based Object Manipulation](https://mlsanity.com/paper/2107.08865/) (2021)

## Abstract (arXiv)

We propose an approach for dense semantic 3D reconstruction which uses a data term that is defined as potentials over viewing rays, combined with continuous surface area penalization. Our formulation is a convex relaxation which we augment with a crucial non-convex constraint that ensures exact handling of visibility. To tackle the non-convex minimization problem, we propose a majorize-minimize type strategy which converges to a critical point. We demonstrate the benefits of using the non-convex constraint experimentally. For the geometry-only case, we set a new state of the art on two datasets of the commonly used Middlebury multi-view stereo benchmark. Moreover, our general-purpose formulation directly reconstructs thin objects, which are usually treated with specialized algorithms. A qualitative evaluation on the dense semantic 3D reconstruction task shows that we improve significantly over previous methods.