# DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models

arXiv: https://arxiv.org/abs/2407.01519 · HTML: https://mlsanity.com/paper/2407.01519/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 92 of its 105 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 30 of 92)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data](https://mlsanity.com/paper/2107.10833/) (2021)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)

## Built on by (most cited first, 30 of 34)

- [SeedVR: Seeding Infinity in Diffusion Transformer Towards Generic Video Restoration](https://mlsanity.com/paper/2501.01320/) (2025)
- [Solving Video Inverse Problems Using Image Diffusion Models](https://mlsanity.com/paper/2409.02574/) (2024)
- [VISION-XL: High Definition Video Inverse Problem Solver using Latent Image Diffusion Models](https://mlsanity.com/paper/2412.00156/) (2024)
- [Show and Polish: Reference-Guided Identity Preservation in Face Video Restoration](https://mlsanity.com/paper/2507.10293/) (2025)
- [LightsOut: Diffusion-based Outpainting for Enhanced Lens Flare Removal](https://mlsanity.com/paper/2510.15868/) (2025)
- [Zero-Shot Video Restoration and Enhancement Using Pre-Trained Image Diffusion Model](https://mlsanity.com/paper/2407.01960/) (2024)
- [Generative AI for Autonomous Driving: Frontiers and Opportunities](https://mlsanity.com/paper/2505.08854/) (2025)
- [Restereo: Diffusion stereo video generation and restoration](https://mlsanity.com/paper/2506.06023/) (2025)
- [Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion](https://mlsanity.com/paper/2512.23709/) (2025)
- [Temporal-Consistent Video Restoration with Pre-trained Diffusion Models](https://mlsanity.com/paper/2503.14863/) (2025)
- [DiTVR: Zero-Shot Diffusion Transformer for Video Restoration](https://mlsanity.com/paper/2508.07811/) (2025)
- [Generative Refocusing: Flexible Defocus Control from a Single Image](https://mlsanity.com/paper/2512.16923/) (2025)
- [STeP: A Framework for Solving Scientific Video Inverse Problems with Spatiotemporal Diffusion Priors](https://mlsanity.com/paper/2504.07549/) (2025)
- [OS-DiffVSR: Towards One-step Latent Diffusion Model for High-detailed Real-world Video Super-Resolution](https://mlsanity.com/paper/2509.16507/) (2025)
- [InstantViR: Real-Time Video Inverse Problem Solver with Distilled Diffusion Prior](https://mlsanity.com/paper/2511.14208/) (2025)
- [LVTINO: LAtent Video consisTency INverse sOlver for High Definition Video Restoration](https://mlsanity.com/paper/2510.01339/) (2025)
- [Rethinking Diffusion Model-Based Video Super-Resolution: Leveraging Dense Guidance from Aligned Features](https://mlsanity.com/paper/2511.16928/) (2025)
- [FIPER: Factorized Features for Robust Image Super-Resolution and Compression](https://mlsanity.com/paper/2410.18083/) (2024)
- [Unsupervised Methods for Video Quality Improvement: A Survey of Restoration and Enhancement Techniques](https://mlsanity.com/paper/2507.08375/) (2025)
- [Semantic and Temporal Integration in Latent Diffusion Space for High-Fidelity Video Super-Resolution](https://mlsanity.com/paper/2508.00471/) (2025)
- [DIPLI: Deep Image Prior Lucky Imaging for Blind Astronomical Image Restoration](https://mlsanity.com/paper/2503.15984/) (2025)
- [MACS: Measurement-Aware Consistency Sampling for Inverse Problems](https://mlsanity.com/paper/2510.02208/) (2025)
- [Improving Temporal Consistency and Fidelity at Inference-time in Perceptual Video Restoration by Zero-shot Image-based Diffusion Models](https://mlsanity.com/paper/2510.25420/) (2025)
- [Zero-Shot Video Restoration and Enhancement with Assistance of Video Diffusion Models](https://mlsanity.com/paper/2601.21922/) (2026)
- [DVFace: Spatio-Temporal Dual-Prior Diffusion for Video Face Restoration](https://mlsanity.com/paper/2604.14560/) (2026)
- [Accelerating Video Inverse Problem Solvers with Autoregressive Diffusion Models](https://mlsanity.com/paper/2605.20624/) (2026)
- [Reflection Separation from a Single Image via Joint Latent Diffusion](https://mlsanity.com/paper/2606.04107/) (2026)
- [TIGER: Taming Identity, Geometry, and Generative Priors for High-Quality Face Video Restoration](https://mlsanity.com/paper/2606.24336/) (2026)
- [LongE2V: Long-Horizon Event-based Video Reconstruction, Prediction, and Frame Interpolation with Video Diffusion Models](https://mlsanity.com/paper/2607.08770/) (2026)
- [4D Human-Scene Reconstruction from Low-Overlap Captures](https://mlsanity.com/paper/2607.09125/) (2026)

## Abstract (arXiv)

We present DiffIR2VR-Zero, a zero-shot framework that enables any pre-trained image restoration diffusion model to perform high-quality video restoration without additional training. While image diffusion models have shown remarkable restoration capabilities, their direct application to video leads to temporal inconsistencies, and existing video restoration methods require extensive retraining for different degradation types. Our approach addresses these challenges through two key innovations: a hierarchical latent warping strategy that maintains consistency across both keyframes and local frames, and a hybrid token merging mechanism that adaptively combines optical flow and feature matching. Through extensive experiments, we demonstrate that our method not only maintains the high-quality restoration of base diffusion models but also achieves superior temporal consistency across diverse datasets and degradation conditions, including challenging scenarios like 8× super-resolution and severe noise. Importantly, our framework works with any image restoration diffusion model, providing a versatile solution for video enhancement without task-specific training or modifications. Project page: https://jimmycv07.github.io/DiffIR2VRweb/