# Temporal Feature Warping for Video Shadow Detection

arXiv: https://arxiv.org/abs/2107.14287 · HTML: https://mlsanity.com/paper/2107.14287/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 29 Jul 2021.

## Builds on (most cited first, 20 of 20)

- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Deep Feature Flow for Video Recognition](https://mlsanity.com/paper/1611.07715/) (2016)
- [ParseNet: Looking Wider to See Better](https://mlsanity.com/paper/1506.04579/) (2015)
- [Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal](https://mlsanity.com/paper/1712.02478/) (2017)
- [Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation](https://mlsanity.com/paper/2003.13045/) (2020)
- [Shadow Removal via Shadow Image Decomposition](https://mlsanity.com/paper/1908.08628/) (2019)
- [Semantic Video CNNs through Representation Warping](https://mlsanity.com/paper/1708.03088/) (2017)
- [From Shadow Segmentation to Shadow Removal](https://mlsanity.com/paper/2008.00267/) (2020)
- [Direction-aware Spatial Context Features for Shadow Detection](https://mlsanity.com/paper/1712.04142/) (2017)
- [Semi-Supervised Video Salient Object Detection Using Pseudo-Labels](https://mlsanity.com/paper/1908.04051/) (2019)
- [ARGAN: Attentive Recurrent Generative Adversarial Network for Shadow Detection and Removal](https://mlsanity.com/paper/1908.01323/) (2019)
- [A+D Net: Training a Shadow Detector with Adversarial Shadow Attenuation](https://mlsanity.com/paper/1712.01361/) (2017)
- [Physics-based Shadow Image Decomposition for Shadow Removal](https://mlsanity.com/paper/2012.13018/) (2020)
- [Revisiting Shadow Detection: A New Benchmark Dataset for Complex World](https://mlsanity.com/paper/1911.06998/) (2019)
- [Triple-cooperative Video Shadow Detection](https://mlsanity.com/paper/2103.06533/) (2021)
- [People as Scene Probes](https://mlsanity.com/paper/2007.09209/) (2020)
- [Shadow Detection: A Survey and Comparative Evaluation of Recent Methods](https://mlsanity.com/paper/1304.1233/) (2013)
- [Geodesic Distance Histogram Feature for Video Segmentation](https://mlsanity.com/paper/1704.00077/) (2017)

## Built on by (most cited first, 5 of 5)

- [SCOTCH and SODA: A Transformer Video Shadow Detection Framework](https://mlsanity.com/paper/2211.06885/) (2022)
- [Detect Any Shadow: Segment Anything for Video Shadow Detection](https://mlsanity.com/paper/2305.16698/) (2023)
- [Learning Shadow Correspondence for Video Shadow Detection](https://mlsanity.com/paper/2208.00150/) (2022)
- [Unveiling Deep Shadows: A Survey and Benchmark on Image and Video Shadow Detection, Removal, and Generation in the Deep Learning Era](https://mlsanity.com/paper/2409.02108/) (2024)
- [When SAM2 Meets Video Shadow and Mirror Detection](https://mlsanity.com/paper/2412.19293/) (2024)

## Abstract (arXiv)

While single image shadow detection has been improving rapidly in recent years, video shadow detection remains a challenging task due to data scarcity and the difficulty in modelling temporal consistency. The current video shadow detection method achieves this goal via co-attention, which mostly exploits information that is temporally coherent but is not robust in detecting moving shadows and small shadow regions. In this paper, we propose a simple but powerful method to better aggregate information temporally. We use an optical flow based warping module to align and then combine features between frames. We apply this warping module across multiple deep-network layers to retrieve information from neighboring frames including both local details and high-level semantic information. We train and test our framework on the ViSha dataset. Experimental results show that our model outperforms the state-of-the-art video shadow detection method by 28%, reducing BER from 16.7 to 12.0.