# UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections

arXiv: https://arxiv.org/abs/2312.13285 · HTML: https://mlsanity.com/paper/2312.13285/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 63 references that are in the map.
- First posted on arXiv 20 Dec 2023.

## Builds on (most cited first, 30 of 49)

- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2112.00724/) (2021)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis](https://mlsanity.com/paper/2012.03927/) (2020)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction](https://mlsanity.com/paper/2206.00665/) (2022)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [Neuralangelo: High-Fidelity Neural Surface Reconstruction](https://mlsanity.com/paper/2306.03092/) (2023)
- [NeRD: Neural Reflectance Decomposition from Image Collections](https://mlsanity.com/paper/2012.03918/) (2020)
- [Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching](https://mlsanity.com/paper/1912.06378/) (2019)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](https://mlsanity.com/paper/2104.00674/) (2021)

## Built on by (most cited first, 30 of 30)

- [EnvGS: Modeling View-Dependent Appearance with Environment Gaussian](https://mlsanity.com/paper/2412.15215/) (2024)
- [NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections](https://mlsanity.com/paper/2405.14871/) (2024)
- [IllumiNeRF: 3D Relighting Without Inverse Rendering](https://mlsanity.com/paper/2406.06527/) (2024)
- [MaterialRefGS: Reflective Gaussian Splatting with Multi-view Consistent Material Inference](https://mlsanity.com/paper/2510.11387/) (2025)
- [MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections](https://mlsanity.com/paper/2405.11921/) (2024)
- [Learning-based Multi-View Stereo: A Survey](https://mlsanity.com/paper/2408.15235/) (2024)
- [PISR: Polarimetric Neural Implicit Surface Reconstruction for Textureless and Specular Objects](https://mlsanity.com/paper/2409.14331/) (2024)
- [Geometry Field Splatting with Gaussian Surfels](https://mlsanity.com/paper/2411.17067/) (2024)
- [MonoGSDF: Exploring Monocular Geometric Cues for Gaussian Splatting-Guided Implicit Surface Reconstruction](https://mlsanity.com/paper/2411.16898/) (2024)
- [GSurf: Learning Signed Distance Fields from Splatting Opaque Gaussians for High-quality 3D Reconstruction](https://mlsanity.com/paper/2411.15723/) (2024)
- [AniSDF: Fused-Granularity Neural Surfaces with Anisotropic Encoding for High-Fidelity 3D Reconstruction](https://mlsanity.com/paper/2410.01202/) (2024)
- [Eye2Eye: A Simple Approach for Monocular-to-Stereo Video Synthesis](https://mlsanity.com/paper/2505.00135/) (2025)
- [Car-GS: Addressing Reflective and Transparent Surface Challenges in 3D Car Reconstruction](https://mlsanity.com/paper/2501.11020/) (2025)
- [Differentiable Light Transport with Gaussian Surfels via Adapted Radiosity for Efficient Relighting and Geometry Reconstruction](https://mlsanity.com/paper/2509.18497/) (2025)
- [EUGens: Efficient, Unified, and General Dense Layers](https://mlsanity.com/paper/2601.22563/) (2026)
- [RT-Splatting: Joint Reflection-Transmission Modeling with Gaussian Splatting](https://mlsanity.com/paper/2605.18263/) (2026)
- [Multi-times Monte Carlo Rendering for Inter-reflection Reconstruction](https://mlsanity.com/paper/2407.05771/) (2024)
- [Structurally Disentangled Feature Fields Distillation for 3D Understanding and Editing](https://mlsanity.com/paper/2502.14789/) (2025)
- [Reconstruction Using the Invisible: Intuition from NIR and Metadata for Enhanced 3D Gaussian Splatting](https://mlsanity.com/paper/2508.14443/) (2025)
- [Light Field Based 6DoF Tracking of Previously Unobserved Objects](https://mlsanity.com/paper/2512.13007/) (2025)
- [Lite3R: A Model-Agnostic Framework for Efficient Feed-Forward 3D Reconstruction](https://mlsanity.com/paper/2605.11354/) (2026)
- [EUGens: Efficient, Unified, and General Dense Layers](https://mlsanity.com/paper/2410.09771/) (2024)
- [TraceFlow: Dynamic 3D Reconstruction of Specular Scenes Driven by Ray Tracing](https://mlsanity.com/paper/2512.10095/) (2025)
- [SSR-GS: Separating Specular Reflection in Gaussian Splatting for Glossy Surface Reconstruction](https://mlsanity.com/paper/2603.05152/) (2026)
- [Ref-DGS: Reflective Dual Gaussian Splatting](https://mlsanity.com/paper/2603.07664/) (2026)
- [PAGaS: Pixel-Aligned 1DoF Gaussian Splatting for Depth Refinement](https://mlsanity.com/paper/2604.22129/) (2026)
- [VEPHand: View-Efficient Photometric Hand Performance Capture at Scale](https://mlsanity.com/paper/2606.15966/) (2026)
- [Reflection-aware Generative Novel View Synthesis](https://mlsanity.com/paper/2609.05382/) (2026)
- [ADELE - Adaptive Delaunay Grids for High-Fidelity Mesh-Native Reconstruction](https://mlsanity.com/paper/2609.06723/) (2026)
- [MicroSDF: Microfacet-Driven Hybrid Neural SDFs for Mixed-Reflectance Surface Reconstruction](https://mlsanity.com/paper/2610.19681/) (2026)

## Abstract (arXiv)

Neural 3D scene representations have shown great potential for 3D reconstruction from 2D images. However, reconstructing real-world captures of complex scenes still remains a challenge. Existing generic 3D reconstruction methods often struggle to represent fine geometric details and do not adequately model reflective surfaces of large-scale scenes. Techniques that explicitly focus on reflective surfaces can model complex and detailed reflections by exploiting better reflection parameterizations. However, we observe that these methods are often not robust in real scenarios where non-reflective as well as reflective components are present. In this work, we propose UniSDF, a general purpose 3D reconstruction method that can reconstruct large complex scenes with reflections. We investigate both camera view as well as reflected view-based color parameterization techniques and find that explicitly blending these representations in 3D space enables reconstruction of surfaces that are more geometrically accurate, especially for reflective surfaces. We further combine this representation with a multi-resolution grid backbone that is trained in a coarse-to-fine manner, enabling faster reconstructions than prior methods. Extensive experiments on object-level datasets DTU, Shiny Blender as well as unbounded datasets Mip-NeRF 360 and Ref-NeRF real demonstrate that our method is able to robustly reconstruct complex large-scale scenes with fine details and reflective surfaces, leading to the best overall performance. Project page: urlhttps://fangjinhuawang.github.io/UniSDF.