# Visiting the Invisible: Layer-by-Layer Completed Scene Decomposition

arXiv: https://arxiv.org/abs/2104.05367 · HTML: https://mlsanity.com/paper/2104.05367/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 71 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 30 of 43)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition](https://mlsanity.com/paper/1406.4729/) (2014)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019)
- [Fully Convolutional Instance-aware Semantic Segmentation](https://mlsanity.com/paper/1611.07709/) (2016)
- [Instance-aware Semantic Segmentation via Multi-task Network Cascades](https://mlsanity.com/paper/1512.04412/) (2015)

## Built on by (most cited first, 23 of 23)

- [Amodal3R: Amodal 3D Reconstruction from Occluded 2D Images](https://mlsanity.com/paper/2503.13439/) (2025)
- [Amodal Ground Truth and Completion in the Wild](https://mlsanity.com/paper/2312.17247/) (2023)
- [Amodal Completion via Progressive Mixed Context Diffusion](https://mlsanity.com/paper/2312.15540/) (2023)
- [MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation](https://mlsanity.com/paper/2404.02790/) (2024)
- [LayerD: Decomposing Raster Graphic Designs into Layers](https://mlsanity.com/paper/2509.25134/) (2025)
- [Open-World Amodal Appearance Completion](https://mlsanity.com/paper/2411.13019/) (2024)
- [Image Amodal Completion: A Survey](https://mlsanity.com/paper/2207.02062/) (2022)
- [Object-level Scene Deocclusion](https://mlsanity.com/paper/2406.07706/) (2024)
- [Visual Jenga: Discovering Object Dependencies via Counterfactual Inpainting](https://mlsanity.com/paper/2503.21770/) (2025)
- [Repositioning the Subject within Image](https://mlsanity.com/paper/2401.16861/) (2024)
- [High-Quality Pluralistic Image Completion via Code Shared VQGAN](https://mlsanity.com/paper/2204.01931/) (2022)
- [Unveiling the Invisible: Reasoning Complex Occlusions Amodally with AURA](https://mlsanity.com/paper/2503.10225/) (2025)
- [See-through: Single-image Layer Decomposition for Anime Characters](https://mlsanity.com/paper/2602.03749/) (2026)
- [Referring Layer Decomposition](https://mlsanity.com/paper/2602.19358/) (2026)
- [SynergyAmodal: Deocclude Anything with Text Control](https://mlsanity.com/paper/2504.19506/) (2025)
- [BBBD: Bounding Box Based Detector for Occlusion Detection and Order Recovery](https://mlsanity.com/paper/2204.12841/) (2022)
- [SynTable: A Synthetic Data Generation Pipeline for Unseen Object Amodal Instance Segmentation of Cluttered Tabletop Scenes](https://mlsanity.com/paper/2307.07333/) (2023)
- [Sequential Amodal Segmentation via Cumulative Occlusion Learning](https://mlsanity.com/paper/2405.05791/) (2024)
- [Shape Distribution Matters: Shape-specific Mixture-of-Experts for Amodal Segmentation under Diverse Occlusions](https://mlsanity.com/paper/2508.01664/) (2025)
- [Stable Diffusion-Based Approach for Human De-Occlusion](https://mlsanity.com/paper/2508.12663/) (2025)
- [Single Point, Full Mask: Velocity-Guided Level Set Evolution for End-to-End Amodal Segmentation](https://mlsanity.com/paper/2508.01661/) (2025)
- [PHAC: Promptable Human Amodal Completion](https://mlsanity.com/paper/2603.14741/) (2026)
- [SemLayer: Semantic-aware Generative Segmentation and Layer Construction for Abstract Icons](https://mlsanity.com/paper/2603.24039/) (2026)

## Abstract (arXiv)

Existing scene understanding systems mainly focus on recognizing the visible parts of a scene, ignoring the intact appearance of physical objects in the real-world. Concurrently, image completion has aimed to create plausible appearance for the invisible regions, but requires a manual mask as input. In this work, we propose a higher-level scene understanding system to tackle both visible and invisible parts of objects and backgrounds in a given scene. Particularly, we built a system to decompose a scene into individual objects, infer their underlying occlusion relationships, and even automatically learn which parts of the objects are occluded that need to be completed. In order to disentangle the occluded relationships of all objects in a complex scene, we use the fact that the front object without being occluded is easy to be identified, detected, and segmented. Our system interleaves the two tasks of instance segmentation and scene completion through multiple iterations, solving for objects layer-by-layer. We first provide a thorough experiment using a new realistically rendered dataset with ground-truths for all invisible regions. To bridge the domain gap to real imagery where ground-truths are unavailable, we then train another model with the pseudo-ground-truths generated from our trained synthesis model. We demonstrate results on a wide variety of datasets and show significant improvement over the state-of-the-art.