# Learning Spatial and Spatio-Temporal Pixel Aggregations for Image and Video Denoising

arXiv: https://arxiv.org/abs/2101.10760 · HTML: https://mlsanity.com/paper/2101.10760/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 54 references that are in the map.
- First posted on arXiv 26 Jan 2021.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising](https://mlsanity.com/paper/1710.04026/) (2017)
- [Learning to See in the Dark](https://mlsanity.com/paper/1805.01934/) (2018)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Dynamic Filter Networks](https://mlsanity.com/paper/1605.09673/) (2016)
- [Benchmarking Denoising Algorithms with Real Photographs](https://mlsanity.com/paper/1707.01313/) (2017)
- [Non-Local Recurrent Network for Image Restoration](https://mlsanity.com/paper/1806.02919/) (2018)
- [Video Frame Interpolation via Adaptive Convolution](https://mlsanity.com/paper/1703.07514/) (2017)
- [Burst Denoising with Kernel Prediction Networks](https://mlsanity.com/paper/1712.02327/) (2017)
- [Neural Nearest Neighbors Networks](https://mlsanity.com/paper/1810.12575/) (2018)
- [Supervised Raw Video Denoising with a Benchmark Dataset on Dynamic Scenes](https://mlsanity.com/paper/2003.14013/) (2020)
- [DVDnet: A Fast Network for Deep Video Denoising](https://mlsanity.com/paper/1906.11890/) (2019)
- [Transformation-Grounded Image Generation Network for Novel 3D View Synthesis](https://mlsanity.com/paper/1703.02921/) (2017)
- [Deep Burst Denoising](https://mlsanity.com/paper/1712.05790/) (2017)
- [Towards Real Scene Super-Resolution with Raw Images](https://mlsanity.com/paper/1905.12156/) (2019)
- [ViDeNN: Deep Blind Video Denoising](https://mlsanity.com/paper/1904.10898/) (2019)
- [Non-Local Color Image Denoising with Convolutional Neural Networks](https://mlsanity.com/paper/1611.06757/) (2016)
- [Iterative Residual CNNs for Burst Photography Applications](https://mlsanity.com/paper/1811.12197/) (2018)

## Built on by (most cited first, 9 of 9)

- [BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment](https://mlsanity.com/paper/2104.13371/) (2021)
- [Video Frame Interpolation Transformer](https://mlsanity.com/paper/2111.13817/) (2021)
- [3D Human Texture Estimation from a Single Image with Transformers](https://mlsanity.com/paper/2109.02563/) (2021)
- [Unidirectional Video Denoising by Mimicking Backward Recurrent Modules with Look-ahead Forward Ones](https://mlsanity.com/paper/2204.05532/) (2022)
- [Recurrent Self-Supervised Video Denoising with Denser Receptive Field](https://mlsanity.com/paper/2308.03608/) (2023)
- [Coarse-to-Fine Video Denoising with Dual-Stage Spatial-Channel Transformer](https://mlsanity.com/paper/2205.00214/) (2022)
- [Towards Real-World Video Denosing: A Practical Video Denosing Dataset and Network](https://mlsanity.com/paper/2207.01356/) (2022)
- [BlazeBVD: Make Scale-Time Equalization Great Again for Blind Video Deflickering](https://mlsanity.com/paper/2403.06243/) (2024)
- [Video Denoising in Fluorescence Guided Surgery](https://mlsanity.com/paper/2411.09798/) (2024)

## Abstract (arXiv)

Existing denoising methods typically restore clear results by aggregating pixels from the noisy input. Instead of relying on hand-crafted aggregation schemes, we propose to explicitly learn this process with deep neural networks. We present a spatial pixel aggregation network and learn the pixel sampling and averaging strategies for image denoising. The proposed model naturally adapts to image structures and can effectively improve the denoised results. Furthermore, we develop a spatio-temporal pixel aggregation network for video denoising to efficiently sample pixels across the spatio-temporal space. Our method is able to solve the misalignment issues caused by large motion in dynamic scenes. In addition, we introduce a new regularization term for effectively training the proposed video denoising model. We present extensive analysis of the proposed method and demonstrate that our model performs favorably against the state-of-the-art image and video denoising approaches on both synthetic and real-world data.