# Video Dynamics Prior: An Internal Learning Approach for Robust Video Enhancements

arXiv: https://arxiv.org/abs/2312.07835 · HTML: https://mlsanity.com/paper/2312.07835/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 53 references that are in the map.
- First posted on arXiv 13 Dec 2023.

## Builds on (most cited first, 30 of 36)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Generative Image Inpainting with Contextual Attention](https://mlsanity.com/paper/1801.07892/) (2018)
- [EDVR: Video Restoration with Enhanced Deformable Convolutional Networks](https://mlsanity.com/paper/1905.02716/) (2019)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment](https://mlsanity.com/paper/2104.13371/) (2021)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- ["Zero-Shot" Super-Resolution using Deep Internal Learning](https://mlsanity.com/paper/1712.06087/) (2017)
- [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [Real-Time Intermediate Flow Estimation for Video Frame Interpolation](https://mlsanity.com/paper/2011.06294/) (2020)
- [Blind Super-Resolution Kernel Estimation using an Internal-GAN](https://mlsanity.com/paper/1909.06581/) (2019)
- [High-Quality Self-Supervised Deep Image Denoising](https://mlsanity.com/paper/1901.10277/) (2019)
- [Frame-Recurrent Video Super-Resolution](https://mlsanity.com/paper/1801.04590/) (2018)
- [Recurrent Back-Projection Network for Video Super-Resolution](https://mlsanity.com/paper/1903.10128/) (2019)
- [FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation](https://mlsanity.com/paper/1907.01361/) (2019)
- [Deep Flow-Guided Video Inpainting](https://mlsanity.com/paper/1905.02884/) (2019)
- ["Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priors](https://mlsanity.com/paper/1812.00467/) (2018)
- [Neural Blind Deconvolution Using Deep Priors](https://mlsanity.com/paper/1908.02197/) (2019)
- [Deep Video Inpainting](https://mlsanity.com/paper/1905.01639/) (2019)
- [Blind Video Temporal Consistency via Deep Video Prior](https://mlsanity.com/paper/2010.11838/) (2020)
- [DVDnet: A Fast Network for Deep Video Denoising](https://mlsanity.com/paper/1906.11890/) (2019)
- [Space-Time-Aware Multi-Resolution Video Enhancement](https://mlsanity.com/paper/2003.13170/) (2020)

## Built on by (most cited first, 5 of 5)

- [Continuous Video Process: Modeling Videos as Continuous Multi-Dimensional Processes for Video Prediction](https://mlsanity.com/paper/2412.04929/) (2024)
- [Video Decomposition Prior: A Methodology to Decompose Videos into Layers](https://mlsanity.com/paper/2412.04930/) (2024)
- [Efficient Continuous Video Flow Model for Video Prediction](https://mlsanity.com/paper/2412.05633/) (2024)
- [Unsupervised Methods for Video Quality Improvement: A Survey of Restoration and Enhancement Techniques](https://mlsanity.com/paper/2507.08375/) (2025)
- [Generative Kaleidoscopic Networks](https://mlsanity.com/paper/2402.11793/) (2024)

## Abstract (arXiv)

In this paper, we present a novel robust framework for low-level vision tasks, including denoising, object removal, frame interpolation, and super-resolution, that does not require any external training data corpus. Our proposed approach directly learns the weights of neural modules by optimizing over the corrupted test sequence, leveraging the spatio-temporal coherence and internal statistics of videos. Furthermore, we introduce a novel spatial pyramid loss that leverages the property of spatio-temporal patch recurrence in a video across the different scales of the video. This loss enhances robustness to unstructured noise in both the spatial and temporal domains. This further results in our framework being highly robust to degradation in input frames and yields state-of-the-art results on downstream tasks such as denoising, object removal, and frame interpolation. To validate the effectiveness of our approach, we conduct qualitative and quantitative evaluations on standard video datasets such as DAVIS, UCF-101, and VIMEO90K-T.