# Memory-guided Image De-raining Using Time-Lapse Data

arXiv: https://arxiv.org/abs/2201.01883 · HTML: https://mlsanity.com/paper/2201.01883/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 73 references that are in the map.
- First posted on arXiv 6 Jan 2022.

## Builds on (most cited first, 30 of 34)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Multi-Stage Progressive Image Restoration](https://mlsanity.com/paper/2102.02808/) (2021)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [Deep Joint Rain Detection and Removal from a Single Image](https://mlsanity.com/paper/1609.07769/) (2016)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [Learning Memory-guided Normality for Anomaly Detection](https://mlsanity.com/paper/2003.13228/) (2020)
- [Progressive Image Deraining Networks: A Better and Simpler Baseline](https://mlsanity.com/paper/1901.09221/) (2019)
- [Multi-Scale Progressive Fusion Network for Single Image Deraining](https://mlsanity.com/paper/2003.10985/) (2020)
- [Density-aware Single Image De-raining using a Multi-stream Dense Network](https://mlsanity.com/paper/1802.07412/) (2018)
- [RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening](https://mlsanity.com/paper/2103.15597/) (2021)
- [Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining](https://mlsanity.com/paper/1807.05698/) (2018)
- [Image De-raining Using a Conditional Generative Adversarial Network](https://mlsanity.com/paper/1701.05957/) (2017)
- [Spatial Attentive Single-Image Deraining with a High Quality Real Rain Dataset](https://mlsanity.com/paper/1904.01538/) (2019)
- [Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering](https://mlsanity.com/paper/1904.04357/) (2019)
- [Clearing the Skies: A deep network architecture for single-image rain removal](https://mlsanity.com/paper/1609.02087/) (2016)
- [A Model-driven Deep Neural Network for Single Image Rain Removal](https://mlsanity.com/paper/2005.01333/) (2020)
- [Video Object Segmentation with Episodic Graph Memory Networks](https://mlsanity.com/paper/2007.07020/) (2020)
- [Kernelized Memory Network for Video Object Segmentation](https://mlsanity.com/paper/2007.08270/) (2020)
- [Single Image Deraining: A Comprehensive Benchmark Analysis](https://mlsanity.com/paper/1903.08558/) (2019)
- [Single Image Deraining: From Model-Based to Data-Driven and Beyond](https://mlsanity.com/paper/1912.07150/) (2019)
- [VITAL: VIsual Tracking via Adversarial Learning](https://mlsanity.com/paper/1804.04273/) (2018)
- [Lightweight Pyramid Networks for Image Deraining](https://mlsanity.com/paper/1805.06173/) (2018)
- [Syn2Real Transfer Learning for Image Deraining using Gaussian Processes](https://mlsanity.com/paper/2006.05580/) (2020)
- [One-shot Learning with Memory-Augmented Neural Networks](https://mlsanity.com/paper/1605.06065/) (2016)
- [Robust Video Content Alignment and Compensation for Rain Removal in a CNN Framework](https://mlsanity.com/paper/1803.10433/) (2018)
- [Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining](https://mlsanity.com/paper/1808.01491/) (2018)
- [High-Resolution Daytime Translation Without Domain Labels](https://mlsanity.com/paper/2003.08791/) (2020)

## Built on by (most cited first, 2 of 2)

- [FPANet: Frequency-based Video Demoireing using Frame-level Post Alignment](https://mlsanity.com/paper/2301.07330/) (2023)
- [Improving Image De-raining Using Reference-Guided Transformers](https://mlsanity.com/paper/2408.00258/) (2024)

## Abstract (arXiv)

This paper addresses the problem of single image de-raining, that is, the task of recovering clean and rain-free background scenes from a single image obscured by a rainy artifact. Although recent advances adopt real-world time-lapse data to overcome the need for paired rain-clean images, they are limited to fully exploit the time-lapse data. The main cause is that, in terms of network architectures, they could not capture long-term rain streak information in the time-lapse data during training owing to the lack of memory components. To address this problem, we propose a novel network architecture based on a memory network that explicitly helps to capture long-term rain streak information in the time-lapse data. Our network comprises the encoder-decoder networks and a memory network. The features extracted from the encoder are read and updated in the memory network that contains several memory items to store rain streak-aware feature representations. With the read/update operation, the memory network retrieves relevant memory items in terms of the queries, enabling the memory items to represent the various rain streaks included in the time-lapse data. To boost the discriminative power of memory features, we also present a novel background selective whitening (BSW) loss for capturing only rain streak information in the memory network by erasing the background information. Experimental results on standard benchmarks demonstrate the effectiveness and superiority of our approach.