# Peeking Behind Objects: Layered Depth Prediction from a Single Image

arXiv: https://arxiv.org/abs/1807.08776 · HTML: https://mlsanity.com/paper/1807.08776/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2018.

## Built on by (most cited first, 23 of 23)

- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [NeX: Real-time View Synthesis with Neural Basis Expansion](https://mlsanity.com/paper/2103.05606/) (2021)
- [Pathdreamer: A World Model for Indoor Navigation](https://mlsanity.com/paper/2105.08756/) (2021)
- [F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories](https://mlsanity.com/paper/2303.15951/) (2023)
- [Geometry-Free View Synthesis: Transformers and no 3D Priors](https://mlsanity.com/paper/2104.07652/) (2021)
- [Behind the Scenes: Density Fields for Single View Reconstruction](https://mlsanity.com/paper/2301.07668/) (2023)
- [Simple and Effective Synthesis of Indoor 3D Scenes](https://mlsanity.com/paper/2204.02960/) (2022)
- [Image Amodal Completion: A Survey](https://mlsanity.com/paper/2207.02062/) (2022)
- [Structural Multiplane Image: Bridging Neural View Synthesis and 3D Reconstruction](https://mlsanity.com/paper/2303.05937/) (2023)
- [Manydepth2: Motion-Aware Self-Supervised Monocular Depth Estimation in Dynamic Scenes](https://mlsanity.com/paper/2312.15268/) (2023)
- [BoostMVSNeRFs: Boosting MVS-based NeRFs to Generalizable View Synthesis in Large-scale Scenes](https://mlsanity.com/paper/2407.15848/) (2024)
- [Multiscale Representation for Real-Time Anti-Aliasing Neural Rendering](https://mlsanity.com/paper/2304.10075/) (2023)
- [Boundary-induced and scene-aggregated network for monocular depth prediction](https://mlsanity.com/paper/2102.13258/) (2021)
- [Amodal Optical Flow](https://mlsanity.com/paper/2311.07761/) (2023)
- [OmniSyn: Synthesizing 360 Videos with Wide-baseline Panoramas](https://mlsanity.com/paper/2202.08752/) (2022)
- [Self-improving Multiplane-to-layer Images for Novel View Synthesis](https://mlsanity.com/paper/2210.01602/) (2022)
- [Virtual Occlusions Through Implicit Depth](https://mlsanity.com/paper/2305.07014/) (2023)
- [MoE3D: A Mixture-of-Experts Module for 3D Reconstruction](https://mlsanity.com/paper/2601.05208/) (2026)
- [RGB-D Image Inpainting Using Generative Adversarial Network with a Late Fusion Approach](https://mlsanity.com/paper/2110.07413/) (2021)
- [PRISM: Progressive Restoration for Scene Graph-based Image Manipulation](https://mlsanity.com/paper/2311.02247/) (2023)
- [DeepDR: Deep Structure-Aware RGB-D Inpainting for Diminished Reality](https://mlsanity.com/paper/2312.00532/) (2023)
- [Illustrator's Depth: Monocular Layer Index Prediction for Image Decomposition](https://mlsanity.com/paper/2511.17454/) (2025)
- [GoToNet: Fast Monocular Scene Exposure and Exploration](https://mlsanity.com/paper/2206.05967/) (2022)

## Abstract (arXiv)

While conventional depth estimation can infer the geometry of a scene from a single RGB image, it fails to estimate scene regions that are occluded by foreground objects. This limits the use of depth prediction in augmented and virtual reality applications, that aim at scene exploration by synthesizing the scene from a different vantage point, or at diminished reality. To address this issue, we shift the focus from conventional depth map prediction to the regression of a specific data representation called Layered Depth Image (LDI), which contains information about the occluded regions in the reference frame and can fill in occlusion gaps in case of small view changes. We propose a novel approach based on Convolutional Neural Networks (CNNs) to jointly predict depth maps and foreground separation masks used to condition Generative Adversarial Networks (GANs) for hallucinating plausible color and depths in the initially occluded areas. We demonstrate the effectiveness of our approach for novel scene view synthesis from a single image.