# Toward Accurate and Temporally Consistent Video Restoration from Raw Data

arXiv: https://arxiv.org/abs/2312.16247 · HTML: https://mlsanity.com/paper/2312.16247/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 62 references that are in the map.
- First posted on arXiv 25 Dec 2023.

## Builds on (most cited first, 30 of 38)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising](https://mlsanity.com/paper/1710.04026/) (2017)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment](https://mlsanity.com/paper/2104.13371/) (2021)
- [BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond](https://mlsanity.com/paper/2012.02181/) (2020)
- [Toward Convolutional Blind Denoising of Real Photographs](https://mlsanity.com/paper/1807.04686/) (2018)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [Unprocessing Images for Learned Raw Denoising](https://mlsanity.com/paper/1811.11127/) (2018)
- [Frame-Recurrent Video Super-Resolution](https://mlsanity.com/paper/1801.04590/) (2018)
- [Non-Local Recurrent Network for Image Restoration](https://mlsanity.com/paper/1806.02919/) (2018)
- [Burst Denoising with Kernel Prediction Networks](https://mlsanity.com/paper/1712.02327/) (2017)
- [Investigating Tradeoffs in Real-World Video Super-Resolution](https://mlsanity.com/paper/2111.12704/) (2021)
- [Video Super-Resolution with Recurrent Structure-Detail Network](https://mlsanity.com/paper/2008.00455/) (2020)
- [Neural Nearest Neighbors Networks](https://mlsanity.com/paper/1810.12575/) (2018)
- [Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN](https://mlsanity.com/paper/1904.10247/) (2019)
- [Video Super-resolution with Temporal Group Attention](https://mlsanity.com/paper/2007.10595/) (2020)
- [Understanding Deformable Alignment in Video Super-Resolution](https://mlsanity.com/paper/2009.07265/) (2020)
- [Supervised Raw Video Denoising with a Benchmark Dataset on Dynamic Scenes](https://mlsanity.com/paper/2003.14013/) (2020)
- [Efficient Video Super-Resolution through Recurrent Latent Space Propagation](https://mlsanity.com/paper/1909.08080/) (2019)
- [Fully Automatic Video Colorization with Self-Regularization and Diversity](https://mlsanity.com/paper/1908.01311/) (2019)
- [Mitigating Artifacts in Real-World Video Super-Resolution Models](https://mlsanity.com/paper/2212.07339/) (2022)
- [Iterative Joint Image Demosaicking and Denoising using a Residual Denoising Network](https://mlsanity.com/paper/1807.06403/) (2018)
- [Video Demoireing with Relation-Based Temporal Consistency](https://mlsanity.com/paper/2204.02957/) (2022)
- [AnimeSR: Learning Real-World Super-Resolution Models for Animation Videos](https://mlsanity.com/paper/2206.07038/) (2022)

## Abstract (arXiv)

Denoising and demosaicking are two fundamental steps in reconstructing a clean full-color video from raw data, while performing video denoising and demosaicking jointly, namely VJDD, could lead to better video restoration performance than performing them separately. In addition to restoration accuracy, another key challenge to VJDD lies in the temporal consistency of consecutive frames. This issue exacerbates when perceptual regularization terms are introduced to enhance video perceptual quality. To address these challenges, we present a new VJDD framework by consistent and accurate latent space propagation, which leverages the estimation of previous frames as prior knowledge to ensure consistent recovery of the current frame. A data temporal consistency (DTC) loss and a relational perception consistency (RPC) loss are accordingly designed. Compared with the commonly used flow-based losses, the proposed losses can circumvent the error accumulation problem caused by inaccurate flow estimation and effectively handle intensity changes in videos, improving much the temporal consistency of output videos while preserving texture details. Extensive experiments demonstrate the leading VJDD performance of our method in term of restoration accuracy, perceptual quality and temporal consistency. Codes and dataset are available at urlhttps://github.com/GuoShi28/VJDD.