# 3D Photography using Context-aware Layered Depth Inpainting

arXiv: https://arxiv.org/abs/2004.04727 · HTML: https://mlsanity.com/paper/2004.04727/

## Key facts

- Cited by 206 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2020.

## Built on by (most cited first, 30 of 206)

- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Dynamic View Synthesis from Dynamic Monocular Video](https://mlsanity.com/paper/2105.06468/) (2021)
- [Depth Pro: Sharp Monocular Metric Depth in Less Than a Second](https://mlsanity.com/paper/2410.02073/) (2024)
- [ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis](https://mlsanity.com/paper/2409.02048/) (2024)
- [Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior](https://mlsanity.com/paper/2303.14184/) (2023)
- [GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image](https://mlsanity.com/paper/2403.12013/) (2024)
- [Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models](https://mlsanity.com/paper/2303.11989/) (2023)
- [WonderWorld: Interactive 3D Scene Generation from a Single Image](https://mlsanity.com/paper/2406.09394/) (2024)
- [Robust Dynamic Radiance Fields](https://mlsanity.com/paper/2301.02239/) (2023)
- [SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image](https://mlsanity.com/paper/2204.00928/) (2022)
- [Efficient Neural Radiance Fields for Interactive Free-viewpoint Video](https://mlsanity.com/paper/2112.01517/) (2021)
- [NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360deg Views](https://mlsanity.com/paper/2211.16431/) (2022)
- [MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis](https://mlsanity.com/paper/2103.14910/) (2021)
- [Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields](https://mlsanity.com/paper/2305.11588/) (2023)
- [SMD-Nets: Stereo Mixture Density Networks](https://mlsanity.com/paper/2104.03866/) (2021)
- [Pathdreamer: A World Model for Indoor Navigation](https://mlsanity.com/paper/2105.08756/) (2021)
- [F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories](https://mlsanity.com/paper/2303.15951/) (2023)
- [Consistent View Synthesis with Pose-Guided Diffusion Models](https://mlsanity.com/paper/2303.17598/) (2023)
- [The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation](https://mlsanity.com/paper/2306.01923/) (2023)
- [Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging](https://mlsanity.com/paper/2105.14021/) (2021)
- [Vision Transformer for NeRF-Based View Synthesis from a Single Input Image](https://mlsanity.com/paper/2207.05736/) (2022)
- [Geometry-Free View Synthesis: Transformers and no 3D Priors](https://mlsanity.com/paper/2104.07652/) (2021)
- [NeRF-SR: High-Quality Neural Radiance Fields using Supersampling](https://mlsanity.com/paper/2112.01759/) (2021)
- [Generative Image Dynamics](https://mlsanity.com/paper/2309.07906/) (2023)
- [4K4D: Real-Time 4D View Synthesis at 4K Resolution](https://mlsanity.com/paper/2310.11448/) (2023)
- [Neural Trajectory Fields for Dynamic Novel View Synthesis](https://mlsanity.com/paper/2105.05994/) (2021)
- [PixelSynth: Generating a 3D-Consistent Experience from a Single Image](https://mlsanity.com/paper/2108.05892/) (2021)
- [InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Images](https://mlsanity.com/paper/2207.11148/) (2022)

## Abstract (arXiv)

We propose a method for converting a single RGB-D input image into a 3D photo - a multi-layer representation for novel view synthesis that contains hallucinated color and depth structures in regions occluded in the original view. We use a Layered Depth Image with explicit pixel connectivity as underlying representation, and present a learning-based inpainting model that synthesizes new local color-and-depth content into the occluded region in a spatial context-aware manner. The resulting 3D photos can be efficiently rendered with motion parallax using standard graphics engines. We validate the effectiveness of our method on a wide range of challenging everyday scenes and show fewer artifacts compared with the state of the arts.