# Detail-Preserving Latent Diffusion for Stable Shadow Removal

arXiv: https://arxiv.org/abs/2412.17630 · HTML: https://mlsanity.com/paper/2412.17630/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 23 Dec 2024.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 33)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Exploiting Diffusion Prior for Real-World Image Super-Resolution](https://mlsanity.com/paper/2305.07015/) (2023)
- [Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction](https://mlsanity.com/paper/2409.18124/) (2024)
- [Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal](https://mlsanity.com/paper/1712.02478/) (2017)
- [StableNormal: Reducing Diffusion Variance for Stable and Sharp Normal](https://mlsanity.com/paper/2406.16864/) (2024)
- [Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN](https://mlsanity.com/paper/1911.08718/) (2019)
- [ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal](https://mlsanity.com/paper/2212.04711/) (2022)
- [Shadow Removal via Shadow Image Decomposition](https://mlsanity.com/paper/1908.08628/) (2019)
- [Mask-ShadowGAN: Learning to Remove Shadows from Unpaired Data](https://mlsanity.com/paper/1903.10683/) (2019)
- [Learning Physics-guided Face Relighting under Directional Light](https://mlsanity.com/paper/1906.03355/) (2019)
- [Refusion: Enabling Large-Size Realistic Image Restoration with Latent-Space Diffusion Models](https://mlsanity.com/paper/2304.08291/) (2023)
- [Auto-Exposure Fusion for Single-Image Shadow Removal](https://mlsanity.com/paper/2103.01255/) (2021)
- [Learning Intrinsic Image Decomposition from Watching the World](https://mlsanity.com/paper/1804.00582/) (2018)
- [Physically-Based Editing of Indoor Scene Lighting from a Single Image](https://mlsanity.com/paper/2205.09343/) (2022)
- [From Shadow Segmentation to Shadow Removal](https://mlsanity.com/paper/2008.00267/) (2020)
- [IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis](https://mlsanity.com/paper/2210.00647/) (2022)
- [DeS3: Adaptive Attention-driven Self and Soft Shadow Removal using ViT Similarity](https://mlsanity.com/paper/2211.08089/) (2022)
- [ShadowFormer: Global Context Helps Image Shadow Removal](https://mlsanity.com/paper/2302.01650/) (2023)
- [Latent Feature-Guided Diffusion Models for Shadow Removal](https://mlsanity.com/paper/2312.02156/) (2023)
- [Recasting Regional Lighting for Shadow Removal](https://mlsanity.com/paper/2402.00341/) (2024)

## Built on by (most cited first, 13 of 13)

- [DenseSR: Image Shadow Removal as Dense Prediction](https://mlsanity.com/paper/2507.16472/) (2025)
- [UniSER: A Foundation Model for Unified Soft Effects Removal](https://mlsanity.com/paper/2511.14183/) (2025)
- [PhaSR: Generalized Image Shadow Removal with Physically Aligned Priors](https://mlsanity.com/paper/2601.17470/) (2026)
- [StableIntrinsic: Detail-preserving One-step Diffusion Model for Multi-view Material Estimation](https://mlsanity.com/paper/2508.19789/) (2025)
- [DeshadowMamba: Deshadowing as 1D Sequential Similarity](https://mlsanity.com/paper/2510.24260/) (2025)
- [CANDLE: Illumination-Invariant Semantic Priors for Color Ambient Lighting Normalization](https://mlsanity.com/paper/2604.02785/) (2026)
- [Winner of CVPR2026 NTIRE Challenge on Image Shadow Removal: Semantic and Geometric Guidance for Shadow Removal via Cascaded Refinement](https://mlsanity.com/paper/2604.16177/) (2026)
- [CFSR: Geometry-Conditioned Shadow Removal via Physical Disentanglement](https://mlsanity.com/paper/2604.18032/) (2026)
- [WildShadowRemover: In-the-Wild Video Shadow Removal via Detail-Preserving Video Diffusion Models](https://mlsanity.com/paper/2607.26203/) (2026)
- [FreeShadow: Training-Free Shadow Removal via Illumination Transfer and Selective Content Preservation in Diffusion Models](https://mlsanity.com/paper/2607.26715/) (2026)
- [Consistency as Regularization for Unsupervised Shadow Removal](https://mlsanity.com/paper/2609.01806/) (2026)
- [After a Decade: Bringing Shadow Removal into the Real World with Agentic Training Data](https://mlsanity.com/paper/2609.38607/) (2026)
- [ODDR: One-Step Deshadow Diffusion via Reward Guidance](https://mlsanity.com/paper/2610.01291/) (2026)

## Abstract (arXiv)

Achieving high-quality shadow removal with strong generalizability is challenging in scenes with complex global illumination. Due to the limited diversity in shadow removal datasets, current methods are prone to overfitting training data, often leading to reduced performance on unseen cases. To address this, we leverage the rich visual priors of a pre-trained Stable Diffusion (SD) model and propose a two-stage fine-tuning pipeline to adapt the SD model for stable and efficient shadow removal. In the first stage, we fix the VAE and fine-tune the denoiser in latent space, which yields substantial shadow removal but may lose some high-frequency details. To resolve this, we introduce a second stage, called the detail injection stage. This stage selectively extracts features from the VAE encoder to modulate the decoder, injecting fine details into the final results. Experimental results show that our method outperforms state-of-the-art shadow removal techniques. The cross-dataset evaluation further demonstrates that our method generalizes effectively to unseen data, enhancing the applicability of shadow removal methods.