# Instance-wise Occlusion and Depth Orders in Natural Scenes

arXiv: https://arxiv.org/abs/2111.14562 · HTML: https://mlsanity.com/paper/2111.14562/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 86 references that are in the map.
- First posted on arXiv 29 Nov 2021.

## Builds on (most cited first, 30 of 61)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition](https://mlsanity.com/paper/1406.4729/) (2014)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [R-FCN: Object Detection via Region-based Fully Convolutional Networks](https://mlsanity.com/paper/1605.06409/) (2016)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018)
- [DeepV2D: Video to Depth with Differentiable Structure from Motion](https://mlsanity.com/paper/1812.04605/) (2018)
- [Single-Image Depth Perception in the Wild](https://mlsanity.com/paper/1604.03901/) (2016)
- [FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation](https://mlsanity.com/paper/1902.09513/) (2019)
- [Attention to Scale: Scale-aware Semantic Image Segmentation](https://mlsanity.com/paper/1511.03339/) (2015)
- [MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning](https://mlsanity.com/paper/2001.06902/) (2020)
- [Feature-metric Loss for Self-supervised Learning of Depth and Egomotion](https://mlsanity.com/paper/2007.10603/) (2020)

## Built on by (most cited first, 30 of 31)

- [Good Seed Makes a Good Crop: Discovering Secret Seeds in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2405.14828/) (2024)
- [Amodal Completion via Progressive Mixed Context Diffusion](https://mlsanity.com/paper/2312.15540/) (2023)
- [MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation](https://mlsanity.com/paper/2404.02790/) (2024)
- [LayerD: Decomposing Raster Graphic Designs into Layers](https://mlsanity.com/paper/2509.25134/) (2025)
- [Open-World Amodal Appearance Completion](https://mlsanity.com/paper/2411.13019/) (2024)
- [Image Amodal Completion: A Survey](https://mlsanity.com/paper/2207.02062/) (2022)
- [ELIP: Enhanced Visual-Language Foundation Models for Image Retrieval](https://mlsanity.com/paper/2502.15682/) (2025)
- [CreatiParser: Generative Image Parsing of Raster Graphic Designs into Editable Layers](https://mlsanity.com/paper/2604.19632/) (2026)
- [UWStereo: A Large Synthetic Dataset for Underwater Stereo Matching](https://mlsanity.com/paper/2409.01782/) (2024)
- [Occ2Net: Robust Image Matching Based on 3D Occupancy Estimation for Occluded Regions](https://mlsanity.com/paper/2308.16160/) (2023)
- [I2E: From Image Pixels to Actionable Interactive Environments for Text-Guided Image Editing](https://mlsanity.com/paper/2601.03741/) (2026)
- [Spotlighting Partially Visible Cinematic Language for Video-to-Audio Generation via Self-distillation](https://mlsanity.com/paper/2507.02271/) (2025)
- [Stable and Consistent Prediction of 3D Characteristic Orientation via Invariant Residual Learning](https://mlsanity.com/paper/2306.11406/) (2023)
- [Putting 3D Spatially Sparse Networks on a Diet](https://mlsanity.com/paper/2112.01316/) (2021)
- [SynergyAmodal: Deocclude Anything with Text Control](https://mlsanity.com/paper/2504.19506/) (2025)
- [Amodal Intra-class Instance Segmentation: Synthetic Datasets and Benchmark](https://mlsanity.com/paper/2303.06596/) (2023)
- [SynTable: A Synthetic Data Generation Pipeline for Unseen Object Amodal Instance Segmentation of Cluttered Tabletop Scenes](https://mlsanity.com/paper/2307.07333/) (2023)
- [DMAT: A Dynamic Mask-Aware Transformer for Human De-occlusion](https://mlsanity.com/paper/2402.04558/) (2024)
- [Sequential Amodal Segmentation via Cumulative Occlusion Learning](https://mlsanity.com/paper/2405.05791/) (2024)
- [Contact-Aware Amodal Completion for Human-Object Interaction via Multi-Regional Inpainting](https://mlsanity.com/paper/2508.00427/) (2025)
- [CObL: Toward Zero-Shot Ordinal Layering without User Prompting](https://mlsanity.com/paper/2508.08498/) (2025)
- [GPT-4 for Occlusion Order Recovery](https://mlsanity.com/paper/2509.22383/) (2025)
- [Holistic Order Prediction in Natural Scenes](https://mlsanity.com/paper/2510.01704/) (2025)
- [Illustrator's Depth: Monocular Layer Index Prediction for Image Decomposition](https://mlsanity.com/paper/2511.17454/) (2025)
- [OcclusionFormer: Arranging Z-Order for Layout-Grounded Image Generation](https://mlsanity.com/paper/2605.21343/) (2026)
- [Occlusion-Ordered Semantic Instance Segmentation](https://mlsanity.com/paper/2504.14054/) (2025)
- [Accuracy Does Not Guarantee Human-Likeness: Cross-Domain Human-Centered Benchmark in Monocular Depth Estimation](https://mlsanity.com/paper/2512.08163/) (2025)
- [Taking Shortcuts for Categorical VQA Using Super Neurons](https://mlsanity.com/paper/2603.10781/) (2026)
- [Integrating Multimodal Large Language Model Knowledge into Amodal Completion](https://mlsanity.com/paper/2603.28333/) (2026)
- [OccluRank: Controllable Occlusion-Aware Layout-to-Image Generation by Adding Just an Ordinal Rank](https://mlsanity.com/paper/2608.20932/) (2026)

## Abstract (arXiv)

In this paper, we introduce a new dataset, named InstaOrder, that can be used to understand the geometrical relationships of instances in an image. The dataset consists of 2.9M annotations of geometric orderings for class-labeled instances in 101K natural scenes. The scenes were annotated by 3,659 crowd-workers regarding (1) occlusion order that identifies occluder/occludee and (2) depth order that describes ordinal relations that consider relative distance from the camera. The dataset provides joint annotation of two kinds of orderings for the same instances, and we discover that the occlusion order and depth order are complementary. We also introduce a geometric order prediction network called InstaOrderNet, which is superior to state-of-the-art approaches. Moreover, we propose a dense depth prediction network called InstaDepthNet that uses auxiliary geometric order loss to boost the accuracy of the state-of-the-art depth prediction approach, MiDaS [56].