mlsanityData to 7 Oct 2026

Warped Diffusion: Solving Video Inverse Problems with Image Diffusion Models

arXiv 2410.16152 ↗First posted 21 Oct 2024cs.CV cs.AI cs.LGcited by 21

On the map: Diffusion models → Inverse problems and image restoration

Abstract (arXiv)

Using image models naively for solving inverse video problems often suffers from flickering, texture-sticking, and temporal inconsistency in generated videos. To tackle these problems, in this paper, we view frames as continuous functions in the 2D space, and videos as a sequence of continuous warping transformations between different frames. This perspective allows us to train function space diffusion models only on images and utilize them to solve temporally correlated inverse problems. The function space diffusion models need to be equivariant with respect to the underlying spatial transformations. To ensure temporal consistency, we introduce a simple post-hoc test-time guidance towards (self)-equivariant solutions. Our method allows us to deploy state-of-the-art latent diffusion models such as Stable Diffusion XL to solve video inverse problems. We demonstrate the effectiveness of our method for video inpainting and 8× video super-resolution, outperforming existing techniques based on noise transformations. We provide generated video results: https://giannisdaras.github.io/warpeddiffusion.github.io/.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models. Builds on 75 of its 86 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

75 of 86 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024A Survey on Diffusion Models for Inverse Problems30 Sep
  2. 2024Ambient Diffusion Posterior Sampling: Solving Inverse Problems with Diffusion Models Trained on Corrupted Data13 Mar
  3. 2023Enhancing Perceptual Quality in Video Super-Resolution through Temporally-Consistent Detail Synthesis using Diffusion Models27 Nov
  4. 2023Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models2 Jul
  5. 2023A Variational Perspective on Solving Inverse Problems with Diffusion Models7 May
  6. 2023Solving Inverse Problems with Score-Based Generative Priors learned from Noisy Data2 May
  7. 2023Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration20 Mar
  8. 2022Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model1 Dec
  9. 2022Diffusion Posterior Sampling for General Noisy Inverse Problems29 Sep
  10. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraints2 Jun
  11. 2022Denoising Diffusion Restoration Models27 Jan
  12. 2021Deblurring via Stochastic Refinement5 Dec

Other references · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2022Scalable Diffusion Models with Transformerscited by 5,931
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
38 more
  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  3. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  4. 2016Improved Techniques for Training GANscited by 2,938
  5. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  6. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  8. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  9. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  10. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  11. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  12. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  13. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  14. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  15. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  16. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 666
  17. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  18. 2021Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modelingcited by 594
  19. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 511
  20. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  21. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  22. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  23. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  24. 2022DiffDock: Diffusion Steps, Twists, and Turns for Molecular Dockingcited by 307
  25. 2021Robust Compressed Sensing MRI with Deep Generative Priorscited by 284
  26. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  27. 2022Cold Diffusion: Inverting Arbitrary Image Transforms Without Noisecited by 275
  28. 2022Torsional Diffusion for Molecular Conformer Generationcited by 262
  29. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  30. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
  31. 2022Diffusion models as plug-and-play priorscited by 230
  32. 2023I2SB: Image-to-Image Schr"odinger Bridgecited by 214
  33. 2023A Survey on Video Diffusion Modelscited by 203
  34. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  35. 2020Efficiently Sampling Functions from Gaussian Process Posteriorscited by 125
  36. 2023FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generationcited by 119
  37. 2023Ambient Diffusion: Learning Clean Distributions from Corrupted Datacited by 108
  38. 2023DiffiT: Diffusion Vision Transformers for Image Generationcited by 104

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 12; 2026: 6.

21 papers.

Sort:

By area:

  1. 2024Video Depth without Video Modelscited by 28
  2. 2025Solving Inverse Problems via Diffusion-Based Priors: An Approximation-Free Ensemble Sampling Approachcited by 18
  3. 2025Alias-Free Latent Diffusion Models: Improving Fractional Shift Equivariance of Diffusion Latent Spacecited by 14
  4. 2024VISION-XL: High Definition Video Inverse Problem Solver using Latent Image Diffusion Modelscited by 13
  5. 2024Infinite-Resolution Integral Noise Warping for Diffusion Modelscited by 5
  6. 2025On Equivariance and Fast Sampling in Video Diffusion Models Trained with Warped Noisecited by 4
  7. 2025STeP: A Framework for Solving Scientific Video Inverse Problems with Spatiotemporal Diffusion Priorscited by 3
  8. 2025InstantViR: Real-Time Video Inverse Problem Solver with Distilled Diffusion Priorcited by 3
  9. 2026CameraNoise: Enabling Faithful Camera Control in Video Diffusion through Geometry-Flow-Guided Noise Warpingcited by 3
  10. 2025High Volume Rate 3D Ultrasound Reconstruction with Diffusion Modelscited by 2
  11. 2025LatentMove: Towards Complex Human Movement Video Generationcited by 1
  12. 2025EquiReg: Equivariance Regularized Diffusion for Inverse Problemscited by 1
9 more
  1. 2025Friction on Demand: A Generative Framework for the Inverse Design of Metainterfacescited by 1
  2. 2025Non-rigid Motion Correction for MRI Reconstruction via Coarse-To-Fine Diffusion Modelscited by 0
  3. 2025MACS: Measurement-Aware Consistency Sampling for Inverse Problemscited by 0
  4. 2025Improving Temporal Consistency and Fidelity at Inference-time in Perceptual Video Restoration by Zero-shot Image-based Diffusion Modelscited by 0
  5. 2026Iterative Amortized Hierarchical VAEcited by 0
  6. 2026SeamCam: Quantifying Seamless Camouflage via Multi-Cue Visual Detectabilitycited by 0
  7. 2026Accelerating Video Inverse Problem Solvers with Autoregressive Diffusion Modelscited by 0
  8. 2026Piecewise Dynamic Diffusion Regularization for Reconstruction of Cardiac Cine MRIcited by 0
  9. 20264D Human-Scene Reconstruction from Low-Overlap Capturescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Solving Video Inverse Problems Using Image Diffusion Models
  2. 2025Rethinking Video Super-Resolution: Towards Diffusion-Based Methods without Motion Alignment
  3. 2025LVTINO: LAtent Video consisTency INverse sOlver for High Definition Video Restoration
  4. 2024G2D2: Gradient-Guided Discrete Diffusion for Inverse Problem Solving
  5. 2025Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Rendering
  6. 2025VideoPDE: Unified Generative PDE Solving via Video Inpainting Diffusion Models
  7. 2024Gaussian is All You Need: A Unified Framework for Solving Inverse Problems via Diffusion Posterior Sampling
  8. 2023Motion-Guided Latent Diffusion for Temporally Consistent Real-world Video Super-resolution