# DepGAN: Leveraging Depth Maps for Handling Occlusions and Transparency in Image Composition

arXiv: https://arxiv.org/abs/2407.11890 · HTML: https://mlsanity.com/paper/2407.11890/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 56 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 30 of 32)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [AnyDoor: Zero-shot Object-level Image Customization](https://mlsanity.com/paper/2307.09481/) (2023)
- [Learning Shape Abstractions by Assembling Volumetric Primitives](https://mlsanity.com/paper/1612.00404/) (2016)
- [ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing](https://mlsanity.com/paper/1803.01837/) (2018)
- [Improving the Harmony of the Composite Image by Spatial-Separated Attention Module](https://mlsanity.com/paper/1907.06406/) (2019)
- [Spatial Fusion GAN for Image Synthesis](https://mlsanity.com/paper/1812.05840/) (2018)
- [IMPRINT: Generative Object Compositing by Learning Identity-Preserving Representation](https://mlsanity.com/paper/2403.10701/) (2024)
- [Making Images Real Again: A Comprehensive Survey on Deep Image Composition](https://mlsanity.com/paper/2106.14490/) (2021)
- [High-Resolution Image Harmonization via Collaborative Dual Transformations](https://mlsanity.com/paper/2109.06671/) (2021)
- [Foreground-aware Semantic Representations for Image Harmonization](https://mlsanity.com/paper/2006.00809/) (2020)
- [Learning to Generate Synthetic Data via Compositing](https://mlsanity.com/paper/1904.05475/) (2019)
- [BargainNet: Background-Guided Domain Translation for Image Harmonization](https://mlsanity.com/paper/2009.09169/) (2020)
- [Unsupervised Layered Image Decomposition into Object Prototypes](https://mlsanity.com/paper/2104.14575/) (2021)
- [ControlCom: Controllable Image Composition using Diffusion Model](https://mlsanity.com/paper/2308.10040/) (2023)
- [Learning to Segment via Cut-and-Paste](https://mlsanity.com/paper/1803.06414/) (2018)
- [Context-Aware Synthesis and Placement of Object Instances](https://mlsanity.com/paper/1812.02350/) (2018)
- [Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization](https://mlsanity.com/paper/2109.05750/) (2021)
- [CustomNet: Zero-shot Object Customization with Variable-Viewpoints in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2310.19784/) (2023)
- [SCS-Co: Self-Consistent Style Contrastive Learning for Image Harmonization](https://mlsanity.com/paper/2204.13962/) (2022)
- [Where and Who? Automatic Semantic-Aware Person Composition](https://mlsanity.com/paper/1706.01021/) (2017)
- [Adversarial Image Composition with Auxiliary Illumination](https://mlsanity.com/paper/2009.08255/) (2020)
- [Paste, Inpaint and Harmonize via Denoising: Subject-Driven Image Editing with Pre-Trained Diffusion Model](https://mlsanity.com/paper/2306.07596/) (2023)
- [On the Importance of Visual Context for Data Augmentation in Scene Understanding](https://mlsanity.com/paper/1809.02492/) (2018)
- [DreamCom: Finetuning Text-guided Inpainting Model for Image Composition](https://mlsanity.com/paper/2309.15508/) (2023)
- [3D-aware Blending with Generative NeRFs](https://mlsanity.com/paper/2302.06608/) (2023)

## Abstract (arXiv)

Image composition is a complex task which requires a lot of information about the scene for an accurate and realistic composition, such as perspective, lighting, shadows, occlusions, and object interactions. Previous methods have predominantly used 2D information for image composition, neglecting the potentials of 3D spatial information. In this work, we propose DepGAN, a Generative Adversarial Network that utilizes depth maps and alpha channels to rectify inaccurate occlusions and enhance transparency effects in image composition. Central to our network is a novel loss function called Depth Aware Loss which quantifies the pixel wise depth difference to accurately delineate occlusion boundaries while compositing objects at different depth levels. Furthermore, we enhance our network's learning process by utilizing opacity data, enabling it to effectively manage compositions involving transparent and semi-transparent objects. We tested our model against state-of-the-art image composition GANs on benchmark (both real and synthetic) datasets. The results reveal that DepGAN significantly outperforms existing methods in terms of accuracy of object placement semantics, transparency and occlusion handling, both visually and quantitatively. Our code is available at https://amrtsg.github.io/DepGAN/.