# Shadow Generation for Composite Image Using Diffusion model

arXiv: https://arxiv.org/abs/2403.15234 · HTML: https://mlsanity.com/paper/2403.15234/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 40 of its 58 references that are in the map.
- First posted on arXiv 22 Mar 2024.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [Pluralistic Image Completion](https://mlsanity.com/paper/1903.04227/) (2019)
- [DoveNet: Deep Image Harmonization via Domain Verification](https://mlsanity.com/paper/1911.13239/) (2019)
- [Deep Image Harmonization](https://mlsanity.com/paper/1703.00069/) (2017)
- [ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal](https://mlsanity.com/paper/2212.04711/) (2022)
- [Mask-ShadowGAN: Learning to Remove Shadows from Unpaired Data](https://mlsanity.com/paper/1903.10683/) (2019)
- [ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing](https://mlsanity.com/paper/1803.01837/) (2018)
- [Deep Parametric Indoor Lighting Estimation](https://mlsanity.com/paper/1910.08812/) (2019)
- [Improving the Harmony of the Composite Image by Spatial-Separated Attention Module](https://mlsanity.com/paper/1907.06406/) (2019)
- [GP-GAN: Towards Realistic High-Resolution Image Blending](https://mlsanity.com/paper/1703.07195/) (2017)
- [Making Images Real Again: A Comprehensive Survey on Deep Image Composition](https://mlsanity.com/paper/2106.14490/) (2021)
- [Deep Image Blending](https://mlsanity.com/paper/1910.11495/) (2019)
- [Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model](https://mlsanity.com/paper/2310.07222/) (2023)
- [High-Resolution Image Harmonization via Collaborative Dual Transformations](https://mlsanity.com/paper/2109.06671/) (2021)
- [BargainNet: Background-Guided Domain Translation for Image Harmonization](https://mlsanity.com/paper/2009.09169/) (2020)
- [CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware Training](https://mlsanity.com/paper/2203.11947/) (2022)
- [ControlCom: Controllable Image Composition using Diffusion Model](https://mlsanity.com/paper/2308.10040/) (2023)
- [All-Weather Deep Outdoor Lighting Estimation](https://mlsanity.com/paper/1906.04909/) (2019)
- [Automatic Scene Inference for 3D Object Compositing](https://mlsanity.com/paper/1912.12297/) (2019)

## Built on by (most cited first, 17 of 17)

- [EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing](https://mlsanity.com/paper/2603.19224/) (2026)
- [Precise Object and Effect Removal with Adaptive Target-Aware Attention](https://mlsanity.com/paper/2505.22636/) (2025)
- [MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generation](https://mlsanity.com/paper/2505.21483/) (2025)
- [Unveiling Deep Shadows: A Survey and Benchmark on Image and Video Shadow Detection, Removal, and Generation in the Deep Learning Era](https://mlsanity.com/paper/2409.02108/) (2024)
- [Diffree: Text-Guided Shape Free Object Inpainting with Diffusion Model](https://mlsanity.com/paper/2407.16982/) (2024)
- [MetaShadow: Object-Centered Shadow Detection, Removal, and Synthesis](https://mlsanity.com/paper/2412.02635/) (2024)
- [Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners](https://mlsanity.com/paper/2605.14709/) (2026)
- [Prompt-Aware Controllable Shadow Removal](https://mlsanity.com/paper/2501.15043/) (2025)
- [Refracting Reality: Generating Images with Realistic Transparent Objects](https://mlsanity.com/paper/2511.17340/) (2025)
- [Preserve Anything: Controllable Image Synthesis with Object Preservation](https://mlsanity.com/paper/2506.22531/) (2025)
- [FastInstShadow: A Simple Query-Based Model for Instance Shadow Detection](https://mlsanity.com/paper/2503.07517/) (2025)
- [Embedding Physical Reasoning into Diffusion-Based Shadow Generation Under the Sun and Sky](https://mlsanity.com/paper/2512.06174/) (2025)
- [Joint Shadow Generation and Relighting via Light-Geometry Interaction Maps](https://mlsanity.com/paper/2602.21820/) (2026)
- [MultiShadow: Multi-Object Shadow Generation for Image Compositing via Diffusion Model](https://mlsanity.com/paper/2603.02743/) (2026)
- [VSDiffusion: Taming Ill-Posed Shadow Generation via Visibility-Constrained Diffusion](https://mlsanity.com/paper/2603.08020/) (2026)
- [Reflection Generation for Composite Image Using Diffusion Model](https://mlsanity.com/paper/2604.02168/) (2026)
- [PlantShade: Predicting Plant Shadows for Lighting-Aware Robotic Agricultural Operation](https://mlsanity.com/paper/2609.21059/) (2026)

## Abstract (arXiv)

In the realm of image composition, generating realistic shadow for the inserted foreground remains a formidable challenge. Previous works have developed image-to-image translation models which are trained on paired training data. However, they are struggling to generate shadows with accurate shapes and intensities, hindered by data scarcity and inherent task complexity. In this paper, we resort to foundation model with rich prior knowledge of natural shadow images. Specifically, we first adapt ControlNet to our task and then propose intensity modulation modules to improve the shadow intensity. Moreover, we extend the small-scale DESOBA dataset to DESOBAv2 using a novel data acquisition pipeline. Experimental results on both DESOBA and DESOBAv2 datasets as well as real composite images demonstrate the superior capability of our model for shadow generation task. The dataset, code, and model are released at https://github.com/bcmi/Object-Shadow-Generation-Dataset-DESOBAv2.