# Latent Feature-Guided Diffusion Models for Shadow Removal

arXiv: https://arxiv.org/abs/2312.02156 · HTML: https://mlsanity.com/paper/2312.02156/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 43 of its 58 references that are in the map.
- First posted on arXiv 4 Dec 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)
- [Wasserstein Auto-Encoders](https://mlsanity.com/paper/1711.01558/) (2017)
- [Perception Prioritized Training of Diffusion Models](https://mlsanity.com/paper/2204.00227/) (2022)
- [Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing](https://mlsanity.com/paper/1903.10145/) (2019)
- [Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal](https://mlsanity.com/paper/1712.02478/) (2017)
- [InfoVAE: Information Maximizing Variational Autoencoders](https://mlsanity.com/paper/1706.02262/) (2017)
- [Lagging Inference Networks and Posterior Collapse in Variational Autoencoders](https://mlsanity.com/paper/1901.05534/) (2019)
- [Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN](https://mlsanity.com/paper/1911.08718/) (2019)
- [ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal](https://mlsanity.com/paper/2212.04711/) (2022)
- [Shadow Removal via Shadow Image Decomposition](https://mlsanity.com/paper/1908.08628/) (2019)
- [Mask-ShadowGAN: Learning to Remove Shadows from Unpaired Data](https://mlsanity.com/paper/1903.10683/) (2019)
- [Improved Variational Autoencoders for Text Modeling using Dilated Convolutions](https://mlsanity.com/paper/1702.08139/) (2017)
- [Auto-Exposure Fusion for Single-Image Shadow Removal](https://mlsanity.com/paper/2103.01255/) (2021)
- [DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised Domain-Classifier Guided Network](https://mlsanity.com/paper/2207.10434/) (2022)
- [From Shadow Segmentation to Shadow Removal](https://mlsanity.com/paper/2008.00267/) (2020)
- [From Shadow Generation to Shadow Removal](https://mlsanity.com/paper/2103.12997/) (2021)

## Built on by (most cited first, 29 of 29)

- [ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and Insertion](https://mlsanity.com/paper/2403.18818/) (2024)
- [The Power of Context: How Multimodality Improves Image Super-Resolution](https://mlsanity.com/paper/2503.14503/) (2025)
- [Detail-Preserving Latent Diffusion for Stable Shadow Removal](https://mlsanity.com/paper/2412.17630/) (2024)
- [Single-Image Shadow Removal Using Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/2407.08865/) (2024)
- [Unveiling Deep Shadows: A Survey and Benchmark on Image and Video Shadow Detection, Removal, and Generation in the Deep Learning Era](https://mlsanity.com/paper/2409.02108/) (2024)
- [S3R-Net: A Single-Stage Approach to Self-Supervised Shadow Removal](https://mlsanity.com/paper/2404.12103/) (2024)
- [Shadow Removal Refinement via Material-Consistent Shadow Edges](https://mlsanity.com/paper/2409.06848/) (2024)
- [DenseSR: Image Shadow Removal as Dense Prediction](https://mlsanity.com/paper/2507.16472/) (2025)
- [FieldNet: Efficient Real-Time Shadow Removal for Enhanced Vision in Field Robotics](https://mlsanity.com/paper/2403.08142/) (2024)
- [ShadowHack: Hacking Shadows via Luminance-Color Divide and Conquer](https://mlsanity.com/paper/2412.02545/) (2024)
- [PhaSR: Generalized Image Shadow Removal with Physically Aligned Priors](https://mlsanity.com/paper/2601.17470/) (2026)
- [Semantic-guided Adversarial Diffusion Model for Self-supervised Shadow Removal](https://mlsanity.com/paper/2407.01104/) (2024)
- [Pre-training Feature Guided Diffusion Model for Speech Enhancement](https://mlsanity.com/paper/2406.07646/) (2024)
- [Prompt-Aware Controllable Shadow Removal](https://mlsanity.com/paper/2501.15043/) (2025)
- [DINOLight: Robust Ambient Light Normalization with Self-supervised Visual Prior Integration](https://mlsanity.com/paper/2603.12579/) (2026)
- [ShadowMamba: State-Space Model with Boundary-Region Selective Scan for Shadow Removal](https://mlsanity.com/paper/2411.03260/) (2024)
- [Controlling the Latent Diffusion Model for Generative Image Shadow Removal via Residual Generation](https://mlsanity.com/paper/2412.02322/) (2024)
- [On the Global Photometric Alignment for Low-Level Vision](https://mlsanity.com/paper/2604.08172/) (2026)
- [Soft-Hard Attention U-Net Model and Benchmark Dataset for Multiscale Image Shadow Removal](https://mlsanity.com/paper/2408.03734/) (2024)
- [Shadow Erosion and Nighttime Adaptability for Camera-Based Automated Driving Applications](https://mlsanity.com/paper/2504.08551/) (2025)
- [Contrast-Prior Enhanced Duality for Mask-Free Shadow Removal](https://mlsanity.com/paper/2507.21949/) (2025)
- [Generative Manifold Distillation: Aligning Restoration Trajectories with Natural Image Prior](https://mlsanity.com/paper/2512.11121/) (2025)
- [CANDLE: Illumination-Invariant Semantic Priors for Color Ambient Lighting Normalization](https://mlsanity.com/paper/2604.02785/) (2026)
- [OmniLight: One Model to Rule All Lighting Conditions](https://mlsanity.com/paper/2604.15170/) (2026)
- [Winner of CVPR2026 NTIRE Challenge on Image Shadow Removal: Semantic and Geometric Guidance for Shadow Removal via Cascaded Refinement](https://mlsanity.com/paper/2604.16177/) (2026)
- [WildShadowRemover: In-the-Wild Video Shadow Removal via Detail-Preserving Video Diffusion Models](https://mlsanity.com/paper/2607.26203/) (2026)
- [FreeShadow: Training-Free Shadow Removal via Illumination Transfer and Selective Content Preservation in Diffusion Models](https://mlsanity.com/paper/2607.26715/) (2026)
- [Consistency as Regularization for Unsupervised Shadow Removal](https://mlsanity.com/paper/2609.01806/) (2026)
- [Shadow Feature Refinement Network: Progressive Feature Refinement based on Knowledge Distillation for Effective Shadow Removal](https://mlsanity.com/paper/2610.05325/) (2026)

## Abstract (arXiv)

Recovering textures under shadows has remained a challenging problem due to the difficulty of inferring shadow-free scenes from shadow images. In this paper, we propose the use of diffusion models as they offer a promising approach to gradually refine the details of shadow regions during the diffusion process. Our method improves this process by conditioning on a learned latent feature space that inherits the characteristics of shadow-free images, thus avoiding the limitation of conventional methods that condition on degraded images only. Additionally, we propose to alleviate potential local optima during training by fusing noise features with the diffusion network. We demonstrate the effectiveness of our approach which outperforms the previous best method by 13% in terms of RMSE on the AISTD dataset. Further, we explore instance-level shadow removal, where our model outperforms the previous best method by 82% in terms of RMSE on the DESOBA dataset.