# Soft-Hard Attention U-Net Model and Benchmark Dataset for Multiscale Image Shadow Removal

arXiv: https://arxiv.org/abs/2408.03734 · HTML: https://mlsanity.com/paper/2408.03734/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 81 references that are in the map.
- First posted on arXiv 7 Aug 2024.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal](https://mlsanity.com/paper/1712.02478/) (2017)
- [Pointwise Convolutional Neural Networks](https://mlsanity.com/paper/1712.05245/) (2017)
- [Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN](https://mlsanity.com/paper/1911.08718/) (2019)
- [ShadowDiffusion: When Degradation Prior Meets Diffusion Model for Shadow Removal](https://mlsanity.com/paper/2212.04711/) (2022)
- [Shadow Removal via Shadow Image Decomposition](https://mlsanity.com/paper/1908.08628/) (2019)
- [Mask-ShadowGAN: Learning to Remove Shadows from Unpaired Data](https://mlsanity.com/paper/1903.10683/) (2019)
- [Auto-Exposure Fusion for Single-Image Shadow Removal](https://mlsanity.com/paper/2103.01255/) (2021)
- [DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised Domain-Classifier Guided Network](https://mlsanity.com/paper/2207.10434/) (2022)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [CANet: A Context-Aware Network for Shadow Removal](https://mlsanity.com/paper/2108.09894/) (2021)
- [Physics-based Shadow Image Decomposition for Shadow Removal](https://mlsanity.com/paper/2012.13018/) (2020)
- [Shadow Removal by a Lightness-Guided Network with Training on Unpaired Data](https://mlsanity.com/paper/2006.15617/) (2020)
- [DeS3: Adaptive Attention-driven Self and Soft Shadow Removal using ViT Similarity](https://mlsanity.com/paper/2211.08089/) (2022)
- [Instance Shadow Detection](https://mlsanity.com/paper/1911.07034/) (2019)
- [High-Resolution Document Shadow Removal via A Large-Scale Real-World Dataset and A Frequency-Aware Shadow Erasing Net](https://mlsanity.com/paper/2308.14221/) (2023)
- [ShadowFormer: Global Context Helps Image Shadow Removal](https://mlsanity.com/paper/2302.01650/) (2023)
- [Revisiting Shadow Detection: A New Benchmark Dataset for Complex World](https://mlsanity.com/paper/1911.06998/) (2019)
- [Latent Feature-Guided Diffusion Models for Shadow Removal](https://mlsanity.com/paper/2312.02156/) (2023)
- [Learning from Synthetic Shadows for Shadow Detection and Removal](https://mlsanity.com/paper/2101.01713/) (2021)
- [Recasting Regional Lighting for Shadow Removal](https://mlsanity.com/paper/2402.00341/) (2024)

## Abstract (arXiv)

Effective shadow removal is pivotal in enhancing the visual quality of images in various applications, ranging from computer vision to digital photography. During the last decades physics and machine learning -based methodologies have been proposed; however, most of them have limited capacity in capturing complex shadow patterns due to restrictive model assumptions, neglecting the fact that shadows usually appear at different scales. Also, current datasets used for benchmarking shadow removal are composed of a limited number of images with simple scenes containing mainly uniform shadows cast by single objects, whereas only a few of them include both manual shadow annotations and paired shadow-free images. Aiming to address all these limitations in the context of natural scene imaging, including urban environments with complex scenes, the contribution of this study is twofold: a) it proposes a novel deep learning architecture, named Soft-Hard Attention U-net (SHAU), focusing on multiscale shadow removal; b) it provides a novel synthetic dataset, named Multiscale Shadow Removal Dataset (MSRD), containing complex shadow patterns of multiple scales, aiming to serve as a privacy-preserving dataset for a more comprehensive benchmarking of future shadow removal methodologies. Key architectural components of SHAU are the soft and hard attention modules, which along with multiscale feature extraction blocks enable effective shadow removal of different scales and intensities. The results demonstrate the effectiveness of SHAU over the relevant state-of-the-art shadow removal methods across various benchmark datasets, improving the Peak Signal-to-Noise Ratio and Root Mean Square Error for the shadow area by 25.1% and 61.3%, respectively.