# Language-Driven Interactive Shadow Detection arXiv: https://arxiv.org/abs/2408.08543 ยท HTML: https://mlsanity.com/paper/2408.08543/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 26 of its 60 references that are in the map. - First posted on arXiv 16 Aug 2024. ## Builds on (most cited first, 26 of 26) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016) - [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016) - [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019) - [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016) - [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023) - [Recurrent Multimodal Interaction for Referring Image Segmentation](https://mlsanity.com/paper/1703.07939/) (2017) - [Cross-Modal Progressive Comprehension for Referring Segmentation](https://mlsanity.com/paper/2105.07175/) (2021) - [From Shadow Segmentation to Shadow Removal](https://mlsanity.com/paper/2008.00267/) (2020) - [RainMamba: Enhanced Locality Learning with State Space Models for Video Deraining](https://mlsanity.com/paper/2407.21773/) (2024) - [Towards Robust Referring Video Object Segmentation with Cyclic Relational Consensus](https://mlsanity.com/paper/2207.01203/) (2022) - [Multiview Detection with Shadow Transformer (and View-Coherent Data Augmentation)](https://mlsanity.com/paper/2108.05888/) (2021) - [Instance Shadow Detection](https://mlsanity.com/paper/1911.07034/) (2019) - [Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval](https://mlsanity.com/paper/2201.09168/) (2022) - [AUFormer: Vision Transformers are Parameter-Efficient Facial Action Unit Detectors](https://mlsanity.com/paper/2403.04697/) (2024) - [Dual-Reference Source-Free Active Domain Adaptation for Nasopharyngeal Carcinoma Tumor Segmentation across Multiple Hospitals](https://mlsanity.com/paper/2309.13401/) (2023) - [Triple-cooperative Video Shadow Detection](https://mlsanity.com/paper/2103.06533/) (2021) - [SOGAN: 3D-Aware Shadow and Occlusion Robust GAN for Makeup Transfer](https://mlsanity.com/paper/2104.10567/) (2021) - [GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing](https://mlsanity.com/paper/2403.05916/) (2024) - [SCOTCH and SODA: A Transformer Video Shadow Detection Framework](https://mlsanity.com/paper/2211.06885/) (2022) - [Unsupervised Portrait Shadow Removal via Generative Priors](https://mlsanity.com/paper/2108.03466/) (2021) - [Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generation](https://mlsanity.com/paper/2306.02651/) (2023) - [Multi-scale Promoted Self-adjusting Correlation Learning for Facial Action Unit Detection](https://mlsanity.com/paper/2308.07770/) (2023) - [Shadow Optimization from Structured Deep Edge Detection](https://mlsanity.com/paper/1505.01589/) (2015) - [SDDNet: Style-guided Dual-layer Disentanglement Network for Shadow Detection](https://mlsanity.com/paper/2308.08935/) (2023) - [Advancing UWF-SLO Vessel Segmentation with Source-Free Active Domain Adaptation and a Novel Multi-Center Dataset](https://mlsanity.com/paper/2406.13645/) (2024) ## Built on by (most cited first, 7 of 7) - [MedGround-R1: Advancing Medical Image Grounding via Spatial-Semantic Rewarded Group Relative Policy Optimization](https://mlsanity.com/paper/2507.02994/) (2025) - [Unveiling Deep Shadows: A Survey and Benchmark on Image and Video Shadow Detection, Removal, and Generation in the Deep Learning Era](https://mlsanity.com/paper/2409.02108/) (2024) - [Timeline and Boundary Guided Diffusion Network for Video Shadow Detection](https://mlsanity.com/paper/2408.11785/) (2024) - [Non-Invasive to Invasive: Enhancing FFA Synthesis from CFP with a Benchmark Dataset and a Novel Network](https://mlsanity.com/paper/2410.14965/) (2024) - [CoC: Chain-of-Cancer based on Cross-Modal Autoregressive Traction for Survival Prediction](https://mlsanity.com/paper/2506.15696/) (2025) - [EventRR: Event Referential Reasoning for Referring Video Object Segmentation](https://mlsanity.com/paper/2508.07171/) (2025) - [Revisiting Shadow Detection from a Vision-Language Perspective](https://mlsanity.com/paper/2605.11771/) (2026) ## Abstract (arXiv) Traditional shadow detectors often identify all shadow regions of static images or video sequences. This work presents the Referring Video Shadow Detection (RVSD), which is an innovative task that rejuvenates the classic paradigm by facilitating the segmentation of particular shadows in videos based on descriptive natural language prompts. This novel RVSD not only achieves segmentation of arbitrary shadow areas of interest based on descriptions (flexibility) but also allows users to interact with visual content more directly and naturally by using natural language prompts (interactivity), paving the way for abundant applications ranging from advanced video editing to virtual reality experiences. To pioneer the RVSD research, we curated a well-annotated RVSD dataset, which encompasses 86 videos and a rich set of 15,011 paired textual descriptions with corresponding shadows. To the best of our knowledge, this dataset is the first one for addressing RVSD. Based on this dataset, we propose a Referring Shadow-Track Memory Network (RSM-Net) for addressing the RVSD task. In our RSM-Net, we devise a Twin-Track Synergistic Memory (TSM) to store intra-clip memory features and hierarchical inter-clip memory features, and then pass these memory features into a memory read module to refine features of the current video frame for referring shadow detection. We also develop a Mixed-Prior Shadow Attention (MSA) to utilize physical priors to obtain a coarse shadow map for learning more visual features by weighting it with the input video frame. Experimental results show that our RSM-Net achieves state-of-the-art performance for RVSD with a notable Overall IOU increase of 4.4%. Our code and dataset are available at https://github.com/whq-xxh/RVSD.