# Feature-Aligned Video Raindrop Removal with Temporal Constraints

arXiv: https://arxiv.org/abs/2205.14574 · HTML: https://mlsanity.com/paper/2205.14574/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 40 references that are in the map.
- First posted on arXiv 29 May 2022.

## Builds on (most cited first, 16 of 16)

- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Attentive Generative Adversarial Network for Raindrop Removal from a Single Image](https://mlsanity.com/paper/1711.10098/) (2017)
- [Deep Flow-Guided Video Inpainting](https://mlsanity.com/paper/1905.02884/) (2019)
- [Flow-edge Guided Video Completion](https://mlsanity.com/paper/2009.01835/) (2020)
- [Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN](https://mlsanity.com/paper/1904.10247/) (2019)
- [Predicting the Driver's Focus of Attention: the DR(eye)VE Project](https://mlsanity.com/paper/1705.03854/) (2017)
- [Onion-Peel Networks for Deep Video Completion](https://mlsanity.com/paper/1908.08718/) (2019)
- [Short-Term and Long-Term Context Aggregation Network for Video Inpainting](https://mlsanity.com/paper/2009.05721/) (2020)
- [FastDeRain: A Novel Video Rain Streak Removal Method Using Directional Gradient Priors](https://mlsanity.com/paper/1803.07487/) (2018)
- [I Can See Clearly Now : Image Restoration via De-Raining](https://mlsanity.com/paper/1901.00893/) (2019)
- [Raindrops on Windshield: Dataset and Lightweight Gradient-Based Detection Algorithm](https://mlsanity.com/paper/2104.05078/) (2021)

## Built on by (most cited first, 7 of 7)

- [RainyScape: Unsupervised Rainy Scene Reconstruction using Decoupled Neural Rendering](https://mlsanity.com/paper/2404.11401/) (2024)
- [Rain Removal from Light Field Images with 4D Convolution and Multi-scale Gaussian Process](https://mlsanity.com/paper/2208.07735/) (2022)
- [Joint Depth Estimation and Mixture of Rain Removal From a Single Image](https://mlsanity.com/paper/2303.17766/) (2023)
- [Towards General and Fast Video Derain via Knowledge Distillation](https://mlsanity.com/paper/2308.05346/) (2023)
- [Passive Non-Line-of-Sight Imaging with Light Transport Modulation](https://mlsanity.com/paper/2312.16014/) (2023)
- [Exploiting Regional Information Transformer for Single Image Deraining](https://mlsanity.com/paper/2402.16033/) (2024)
- [Deformable Feature Alignment and Refinement for Moving Infrared Dim-small Target Detection](https://mlsanity.com/paper/2407.07289/) (2024)

## Abstract (arXiv)

Existing adherent raindrop removal methods focus on the detection of the raindrop locations, and then use inpainting techniques or generative networks to recover the background behind raindrops. Yet, as adherent raindrops are diverse in sizes and appearances, the detection is challenging for both single image and video. Moreover, unlike rain streaks, adherent raindrops tend to cover the same area in several frames. Addressing these problems, our method employs a two-stage video-based raindrop removal method. The first stage is the single image module, which generates initial clean results. The second stage is the multiple frame module, which further refines the initial results using temporal constraints, namely, by utilizing multiple input frames in our process and applying temporal consistency between adjacent output frames. Our single image module employs a raindrop removal network to generate initial raindrop removal results, and create a mask representing the differences between the input and initial output. Once the masks and initial results for consecutive frames are obtained, our multiple-frame module aligns the frames in both the image and feature levels and then obtains the clean background. Our method initially employs optical flow to align the frames, and then utilizes deformable convolution layers further to achieve feature-level frame alignment. To remove small raindrops and recover correct backgrounds, a target frame is predicted from adjacent frames. A series of unsupervised losses are proposed so that our second stage, which is the video raindrop removal module, can self-learn from video data without ground truths. Experimental results on real videos demonstrate the state-of-art performance of our method both quantitatively and qualitatively.