# Recurrent Video Deblurring with Blur-Invariant Motion Estimation and Pixel Volumes

arXiv: https://arxiv.org/abs/2108.09982 · HTML: https://mlsanity.com/paper/2108.09982/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 50 references that are in the map.
- First posted on arXiv 23 Aug 2021.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks](https://mlsanity.com/paper/1711.07064/) (2017)
- [Scale-recurrent Network for Deep Image Deblurring](https://mlsanity.com/paper/1802.01770/) (2018)
- [StructureFlow: Image Inpainting via Structure-aware Appearance Flow](https://mlsanity.com/paper/1908.03852/) (2019)
- [Online Video Deblurring via Dynamic Temporal Blending Network](https://mlsanity.com/paper/1704.03285/) (2017)
- [Generalized Video Deblurring for Dynamic Scenes](https://mlsanity.com/paper/1507.02438/) (2015)
- [Video Deblurring via Semantic Segmentation and Pixel-Wise Non-Linear Kernel](https://mlsanity.com/paper/1708.03423/) (2017)
- [Hand-held Video Deblurring via Efficient Fourier Aggregation](https://mlsanity.com/paper/1509.05251/) (2015)
- [Deep Video Deblurring: The Devil is in the Details](https://mlsanity.com/paper/1909.12196/) (2019)
- [Image Super-Resolution via Deterministic-Stochastic Synthesis and Local Statistical Rectification](https://mlsanity.com/paper/1809.06557/) (2018)

## Built on by (most cited first, 30 of 41)

- [VRT: A Video Restoration Transformer](https://mlsanity.com/paper/2201.12288/) (2022)
- [Deblur-NeRF: Neural Radiance Fields from Blurry Images](https://mlsanity.com/paper/2111.14292/) (2021)
- [Recurrent Video Restoration Transformer with Guided Deformable Attention](https://mlsanity.com/paper/2206.02146/) (2022)
- [FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation](https://mlsanity.com/paper/2303.01237/) (2023)
- [BAD-NeRF: Bundle Adjusted Deblur Neural Radiance Fields](https://mlsanity.com/paper/2211.12853/) (2022)
- [DP-NeRF: Deblurred Neural Radiance Field with Physical Scene Priors](https://mlsanity.com/paper/2211.12046/) (2022)
- [A Simple Baseline for Video Restoration with Grouped Spatial-temporal Shift](https://mlsanity.com/paper/2206.10810/) (2022)
- [Improving Image Restoration by Revisiting Global Information Aggregation](https://mlsanity.com/paper/2112.04491/) (2021)
- [Learning Degradation Representations for Image Deblurring](https://mlsanity.com/paper/2208.05244/) (2022)
- [Mitigating Motion Blur in Neural Radiance Fields with Events and Frames](https://mlsanity.com/paper/2403.19780/) (2024)
- [Blur Interpolation Transformer for Real-World Motion from Blur](https://mlsanity.com/paper/2211.11423/) (2022)
- [Unidirectional Video Denoising by Mimicking Backward Recurrent Modules with Look-ahead Forward Ones](https://mlsanity.com/paper/2204.05532/) (2022)
- [Spatio-Temporal Deformable Attention Network for Video Deblurring](https://mlsanity.com/paper/2207.10852/) (2022)
- [Face Deblurring using Dual Camera Fusion on Mobile Phones](https://mlsanity.com/paper/2207.11617/) (2022)
- [Hybrid Neural Rendering for Large-Scale Scenes with Motion Blur](https://mlsanity.com/paper/2304.12652/) (2023)
- [Blur-aware Spatio-temporal Sparse Transformer for Video Deblurring](https://mlsanity.com/paper/2406.07551/) (2024)
- [Efficient Video Deblurring Guided by Motion Magnitude](https://mlsanity.com/paper/2207.13374/) (2022)
- [VDTR: Video Deblurring with Transformer](https://mlsanity.com/paper/2204.08023/) (2022)
- [Deblurring Neural Radiance Fields with Event-driven Bundle Adjustment](https://mlsanity.com/paper/2406.14360/) (2024)
- [PDRF: Progressively Deblurring Radiance Field for Fast and Robust Scene Reconstruction from Blurry Images](https://mlsanity.com/paper/2208.08049/) (2022)
- [Learning Truncated Causal History Model for Video Restoration](https://mlsanity.com/paper/2410.03936/) (2024)
- [Deblur e-NeRF: NeRF from Motion-Blurred Events under High-speed or Low-light Conditions](https://mlsanity.com/paper/2409.17988/) (2024)
- [SMURF: Continuous Dynamics for Motion-Deblurring Radiance Fields](https://mlsanity.com/paper/2403.07547/) (2024)
- [Motion-Aware Animatable Gaussian Avatars Deblurring](https://mlsanity.com/paper/2411.16758/) (2024)
- [Cross-Consistent Deep Unfolding Network for Adaptive All-In-One Video Restoration](https://mlsanity.com/paper/2309.01627/) (2023)
- [Real-Time Video Deblurring via Lightweight Motion Compensation](https://mlsanity.com/paper/2205.12634/) (2022)
- [Deformable Convolutions and LSTM-based Flexible Event Frame Fusion Network for Motion Deblurring](https://mlsanity.com/paper/2306.00834/) (2023)
- [Contribution-based Low-Rank Adaptation with Pre-training Model for Real Image Restoration](https://mlsanity.com/paper/2408.01099/) (2024)
- [DAVIDE: Depth-Aware Video Deblurring](https://mlsanity.com/paper/2409.01274/) (2024)
- [VJT: A Video Transformer on Joint Tasks of Deblurring, Low-light Enhancement and Denoising](https://mlsanity.com/paper/2401.14754/) (2024)

## Abstract (arXiv)

For the success of video deblurring, it is essential to utilize information from neighboring frames. Most state-of-the-art video deblurring methods adopt motion compensation between video frames to aggregate information from multiple frames that can help deblur a target frame. However, the motion compensation methods adopted by previous deblurring methods are not blur-invariant, and consequently, their accuracy is limited for blurry frames with different blur amounts. To alleviate this problem, we propose two novel approaches to deblur videos by effectively aggregating information from multiple video frames. First, we present blur-invariant motion estimation learning to improve motion estimation accuracy between blurry frames. Second, for motion compensation, instead of aligning frames by warping with estimated motions, we use a pixel volume that contains candidate sharp pixels to resolve motion estimation errors. We combine these two processes to propose an effective recurrent video deblurring network that fully exploits deblurred previous frames. Experiments show that our method achieves the state-of-the-art performance both quantitatively and qualitatively compared to recent methods that use deep learning.