# Multi-Stage Raw Video Denoising with Adversarial Loss and Gradient Mask

arXiv: https://arxiv.org/abs/2103.02861 · HTML: https://mlsanity.com/paper/2103.02861/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 69 references that are in the map.
- First posted on arXiv 4 Mar 2021.

## Builds on (most cited first, 30 of 39)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Learning to See in the Dark](https://mlsanity.com/paper/1805.01934/) (2018)
- [Toward Convolutional Blind Denoising of Real Photographs](https://mlsanity.com/paper/1807.04686/) (2018)
- [Unprocessing Images for Learned Raw Denoising](https://mlsanity.com/paper/1811.11127/) (2018)
- [Geometric GAN](https://mlsanity.com/paper/1705.02894/) (2017)
- [Photographic Image Synthesis with Cascaded Refinement Networks](https://mlsanity.com/paper/1707.09405/) (2017)
- [CycleISP: Real Image Restoration via Improved Data Synthesis](https://mlsanity.com/paper/2003.07761/) (2020)
- [Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration](https://mlsanity.com/paper/1508.02848/) (2015)
- [Burst Denoising with Kernel Prediction Networks](https://mlsanity.com/paper/1712.02327/) (2017)
- [FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation](https://mlsanity.com/paper/1907.01361/) (2019)
- [Neural Nearest Neighbors Networks](https://mlsanity.com/paper/1810.12575/) (2018)
- [Multi-Channel Attention Selection GAN with Cascaded Semantic Guidance for Cross-View Image Translation](https://mlsanity.com/paper/1904.06807/) (2019)
- [To learn image super-resolution, use a GAN to learn how to do image degradation first](https://mlsanity.com/paper/1807.11458/) (2018)

## Built on by (most cited first, 4 of 4)

- [RViDeformer: Efficient Raw Video Denoising Transformer with a Larger Benchmark Dataset](https://mlsanity.com/paper/2305.00767/) (2023)
- [Image Denoising Using Green Channel Prior](https://mlsanity.com/paper/2408.05923/) (2024)
- [Spatiotemporal Blind-Spot Network with Calibrated Flow Alignment for Self-Supervised Video Denoising](https://mlsanity.com/paper/2412.11820/) (2024)
- [Combining Pre- and Post-Demosaicking Noise Removal for RAW Video](https://mlsanity.com/paper/2410.02572/) (2024)

## Abstract (arXiv)

In this paper, we propose a learning-based approach for denoising raw videos captured under low lighting conditions. We propose to do this by first explicitly aligning the neighboring frames to the current frame using a convolutional neural network (CNN). We then fuse the registered frames using another CNN to obtain the final denoised frame. To avoid directly aligning the temporally distant frames, we perform the two processes of alignment and fusion in multiple stages. Specifically, at each stage, we perform the denoising process on three consecutive input frames to generate the intermediate denoised frames which are then passed as the input to the next stage. By performing the process in multiple stages, we can effectively utilize the information of neighboring frames without directly aligning the temporally distant frames. We train our multi-stage system using an adversarial loss with a conditional discriminator. Specifically, we condition the discriminator on a soft gradient mask to prevent introducing high-frequency artifacts in smooth regions. We show that our system is able to produce temporally coherent videos with realistic details. Furthermore, we demonstrate through extensive experiments that our approach outperforms state-of-the-art image and video denoising methods both numerically and visually.