# Amodal Ground Truth and Completion in the Wild

arXiv: https://arxiv.org/abs/2312.17247 · HTML: https://mlsanity.com/paper/2312.17247/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 58 references that are in the map.
- First posted on arXiv 28 Dec 2023.

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [Cascade R-CNN: High Quality Object Detection and Instance Segmentation](https://mlsanity.com/paper/1906.09756/) (2019)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [The VIA Annotation Software for Images, Audio and Video](https://mlsanity.com/paper/1904.10699/) (2019)
- [MagicPony: Learning Articulated 3D Animals in the Wild](https://mlsanity.com/paper/2211.12497/) (2022)
- [Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers](https://mlsanity.com/paper/2103.12340/) (2021)
- [Semantic Amodal Segmentation](https://mlsanity.com/paper/1509.01329/) (2015)
- [Self-Supervised Scene De-occlusion](https://mlsanity.com/paper/2004.02788/) (2020)
- [Learning to Segment Object Candidates](https://mlsanity.com/paper/1506.06204/) (2015)
- [Shape Completion Enabled Robotic Grasping](https://mlsanity.com/paper/1609.08546/) (2016)
- [Amodal Instance Segmentation](https://mlsanity.com/paper/1604.08202/) (2016)
- [Segmenting Moving Objects via an Object-Centric Layered Representation](https://mlsanity.com/paper/2207.02206/) (2022)
- [Learning to See the Invisible: End-to-End Trainable Amodal Instance Segmentation](https://mlsanity.com/paper/1804.08864/) (2018)
- [SeGAN: Segmenting and Generating the Invisible](https://mlsanity.com/paper/1703.10239/) (2017)
- [Category-Specific Object Reconstruction from a Single Image](https://mlsanity.com/paper/1411.6069/) (2014)
- [Amodal Segmentation Based on Visible Region Segmentation and Shape Prior](https://mlsanity.com/paper/2012.05598/) (2020)
- [Amodal Panoptic Segmentation](https://mlsanity.com/paper/2202.11542/) (2022)
- [Amodal Completion via Progressive Mixed Context Diffusion](https://mlsanity.com/paper/2312.15540/) (2023)
- [AISFormer: Amodal Instance Segmentation with Transformer](https://mlsanity.com/paper/2210.06323/) (2022)
- [A General Protocol to Probe Large Vision Models for 3D Physical Understanding](https://mlsanity.com/paper/2310.06836/) (2023)

## Built on by (most cited first, 30 of 45)

- [InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environment](https://mlsanity.com/paper/2406.04882/) (2024)
- [pix2gestalt: Amodal Segmentation by Synthesizing Wholes](https://mlsanity.com/paper/2401.14398/) (2024)
- [Amodal3R: Amodal 3D Reconstruction from Occluded 2D Images](https://mlsanity.com/paper/2503.13439/) (2025)
- [When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models](https://mlsanity.com/paper/2405.10255/) (2024)
- [Open-World Amodal Appearance Completion](https://mlsanity.com/paper/2411.13019/) (2024)
- [HiScene: Creating Hierarchical 3D Scenes with Isometric View Generation](https://mlsanity.com/paper/2504.13072/) (2025)
- [Object-level Scene Deocclusion](https://mlsanity.com/paper/2406.07706/) (2024)
- [Using Diffusion Priors for Video Amodal Segmentation](https://mlsanity.com/paper/2412.04623/) (2024)
- [Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactions](https://mlsanity.com/paper/2503.15898/) (2025)
- [LaRender: Training-Free Occlusion Control in Image Generation via Latent Rendering](https://mlsanity.com/paper/2508.07647/) (2025)
- [MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes](https://mlsanity.com/paper/2412.11457/) (2024)
- [TACO: Taming Diffusion for in-the-wild Video Amodal Completion](https://mlsanity.com/paper/2503.12049/) (2025)
- [DeOcc-1-to-3: 3D De-Occlusion from a Single Image via Self-Supervised Multi-View Diffusion](https://mlsanity.com/paper/2506.21544/) (2025)
- [Segment Anything, Even Occluded](https://mlsanity.com/paper/2503.06261/) (2025)
- [Inferring Dynamic Physical Properties from Video Foundation Models](https://mlsanity.com/paper/2510.02311/) (2025)
- [GENA3D: Generative Amodal 3D Modeling by Bridging 2D Priors and 3D Coherence](https://mlsanity.com/paper/2511.21945/) (2025)
- [Amodal Instance Segmentation with Diffusion Shape Prior Estimation](https://mlsanity.com/paper/2409.18256/) (2024)
- [Amodal Depth Anything: Amodal Depth Estimation in the Wild](https://mlsanity.com/paper/2412.02336/) (2024)
- [EraseDraw: Learning to Draw Step-by-Step via Erasing Objects from Images](https://mlsanity.com/paper/2409.00522/) (2024)
- [SeeThrough3D: Occlusion Aware 3D Control in Text-to-Image Generation](https://mlsanity.com/paper/2602.23359/) (2026)
- [Unveiling the Invisible: Reasoning Complex Occlusions Amodally with AURA](https://mlsanity.com/paper/2503.10225/) (2025)
- [EGM: Efficient Visual Grounding Language Models](https://mlsanity.com/paper/2601.13633/) (2026)
- [SynergyAmodal: Deocclude Anything with Text Control](https://mlsanity.com/paper/2504.19506/) (2025)
- [CompleteMe: Reference-based Human Image Completion](https://mlsanity.com/paper/2504.20042/) (2025)
- [Amodal SAM: A Unified Amodal Segmentation Framework with Generalization](https://mlsanity.com/paper/2604.20748/) (2026)
- [EscherNet++: Simultaneous Amodal Completion and Scalable View Synthesis through Masked Fine-Tuning and Enhanced Feed-Forward 3D Reconstruction](https://mlsanity.com/paper/2507.07410/) (2025)
- [Contact-Aware Amodal Completion for Human-Object Interaction via Multi-Regional Inpainting](https://mlsanity.com/paper/2508.00427/) (2025)
- [Shape Distribution Matters: Shape-specific Mixture-of-Experts for Amodal Segmentation under Diverse Occlusions](https://mlsanity.com/paper/2508.01664/) (2025)
- [CObL: Toward Zero-Shot Ordinal Layering without User Prompting](https://mlsanity.com/paper/2508.08498/) (2025)
- [Stable Diffusion-Based Approach for Human De-Occlusion](https://mlsanity.com/paper/2508.12663/) (2025)

## Abstract (arXiv)

This paper studies amodal image segmentation: predicting entire object segmentation masks including both visible and invisible (occluded) parts. In previous work, the amodal segmentation ground truth on real images is usually predicted by manual annotaton and thus is subjective. In contrast, we use 3D data to establish an automatic pipeline to determine authentic ground truth amodal masks for partially occluded objects in real images. This pipeline is used to construct an amodal completion evaluation benchmark, MP3D-Amodal, consisting of a variety of object categories and labels. To better handle the amodal completion task in the wild, we explore two architecture variants: a two-stage model that first infers the occluder, followed by amodal mask completion; and a one-stage model that exploits the representation power of Stable Diffusion for amodal segmentation across many categories. Without bells and whistles, our method achieves a new state-of-the-art performance on Amodal segmentation datasets that cover a large variety of objects, including COCOA and our new MP3D-Amodal dataset. The dataset, model, and code are available at https://www.robots.ox.ac.uk/~vgg/research/amodal/.