# VSDiffusion: Taming Ill-Posed Shadow Generation via Visibility-Constrained Diffusion

arXiv: https://arxiv.org/abs/2603.08020 · HTML: https://mlsanity.com/paper/2603.08020/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Builds on (most cited first, 30 of 30)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [AnyDoor: Zero-shot Object-level Image Customization](https://mlsanity.com/paper/2307.09481/) (2023)
- [DoveNet: Deep Image Harmonization via Domain Verification](https://mlsanity.com/paper/1911.13239/) (2019)
- [ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal](https://mlsanity.com/paper/2212.04711/) (2022)
- [Mask-ShadowGAN: Learning to Remove Shadows from Unpaired Data](https://mlsanity.com/paper/1903.10683/) (2019)
- [GP-GAN: Towards Realistic High-Resolution Image Blending](https://mlsanity.com/paper/1703.07195/) (2017)
- [Making Images Real Again: A Comprehensive Survey on Deep Image Composition](https://mlsanity.com/paper/2106.14490/) (2021)
- [LightIt: Illumination Modeling and Control for Diffusion Models](https://mlsanity.com/paper/2403.10615/) (2024)
- [Putting People in Their Place: Affordance-Aware Human Insertion into Scenes](https://mlsanity.com/paper/2304.14406/) (2023)
- [A Fourier Space Perspective on Diffusion Models](https://mlsanity.com/paper/2505.11278/) (2025)
- [All-Weather Deep Outdoor Lighting Estimation](https://mlsanity.com/paper/1906.04909/) (2019)
- [Shadow Generation for Composite Image in Real-world Scenes](https://mlsanity.com/paper/2104.10338/) (2021)
- [Collage Diffusion](https://mlsanity.com/paper/2303.00262/) (2023)
- [OPA: Object Placement Assessment Dataset](https://mlsanity.com/paper/2107.01889/) (2021)
- [SSN: Soft Shadow Network for Image Compositing](https://mlsanity.com/paper/2007.08211/) (2020)
- [Controllable Shadow Generation Using Pixel Height Maps](https://mlsanity.com/paper/2207.05385/) (2022)
- [Dense Pixel-to-Pixel Harmonization via Continuous Image Representation](https://mlsanity.com/paper/2303.01681/) (2023)
- [Outdoor inverse rendering from a single image using multiview self-supervision](https://mlsanity.com/paper/2102.06591/) (2021)
- [Shadow Generation for Composite Image Using Diffusion model](https://mlsanity.com/paper/2403.15234/) (2024)
- [PixHt-Lab: Pixel Height Based Light Effect Generation for Image Compositing](https://mlsanity.com/paper/2303.00137/) (2023)
- [Fast Object Placement Assessment](https://mlsanity.com/paper/2205.14280/) (2022)
- [Hierarchical Dynamic Image Harmonization](https://mlsanity.com/paper/2211.08639/) (2022)
- [Shadow Generation with Decomposed Mask Prediction and Attentive Shadow Filling](https://mlsanity.com/paper/2306.17358/) (2023)
- [Unveiling Deep Shadows: A Survey and Benchmark on Image and Video Shadow Detection, Removal, and Generation in the Deep Learning Era](https://mlsanity.com/paper/2409.02108/) (2024)

## Abstract (arXiv)

Generating realistic cast shadows for inserted foreground objects is a crucial yet challenging problem in image composition, where maintaining geometric consistency of shadow and object in complex scenes remains difficult due to the ill-posed nature of shadow formation. To address this issue, we propose VSDiffusion, a visibility-constrained two-stage framework designed to narrow the solution space by incorporating visibility priors. In Stage I, we predict a coarse shadow mask to localize plausible shadow generated regions. And in Stage II, conditional diffusion is performed guided by lighting and depth cues estimated from the composite to generate accurate shadows. In VSDiffusion, we inject visibility priors through two complementary pathways. First, a visibility control branch with shadow-gated cross attention that provides multi-scale structural guidance. Then, a learned soft prior map that reweights training loss in error-prone regions to enhance geometric correction. Additionally, we also introduce high-frequency guided enhancement module to sharpen boundaries and improve texture interaction with the background. Experiments on widely used public DESOBAv2 dataset demonstrated that our proposed VSDiffusion can generate accurate shadow, and establishes new SOTA results across most evaluation metrics.