# DreamMover: Leveraging the Prior of Diffusion Models for Image Interpolation with Large Motion arXiv: https://arxiv.org/abs/2409.09605 ยท HTML: https://mlsanity.com/paper/2409.09605/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. - Builds on 50 of its 54 references that are in the map. - First posted on arXiv 15 Sep 2024. ## Placed in - Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/) ## Builds on (most cited first, 30 of 50) - [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021) - [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018) - [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020) - [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020) - [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022) - [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023) - [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022) - [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022) - [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020) - [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021) - [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022) - [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021) - [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022) - [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022) - [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022) - [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023) - [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023) - [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023) - [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017) - [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023) - [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023) - [Emergent Correspondence from Image Diffusion](https://mlsanity.com/paper/2306.03881/) (2023) - [Diffusion Self-Guidance for Controllable Image Generation](https://mlsanity.com/paper/2306.00986/) (2023) - [A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence](https://mlsanity.com/paper/2305.15347/) (2023) - [DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing](https://mlsanity.com/paper/2306.14435/) (2023) - [Softmax Splatting for Video Frame Interpolation](https://mlsanity.com/paper/2003.05534/) (2020) - [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018) ## Built on by (most cited first, 16 of 16) - [Generative Inbetweening through Frame-wise Conditions-Driven Video Generation](https://mlsanity.com/paper/2412.11755/) (2024) - [PERSE: Personalized 3D Generative Avatars from A Single Portrait](https://mlsanity.com/paper/2412.21206/) (2024) - [Alias-Free Latent Diffusion Models: Improving Fractional Shift Equivariance of Diffusion Latent Space](https://mlsanity.com/paper/2503.09419/) (2025) - [AceVFI: A Comprehensive Survey of Advances in Video Frame Interpolation](https://mlsanity.com/paper/2506.01061/) (2025) - [In-2-4D: Inbetweening from Two Single-View Images to 4D Generation](https://mlsanity.com/paper/2504.08366/) (2025) - [TLB-VFI: Temporal-Aware Latent Brownian Bridge Diffusion for Video Frame Interpolation](https://mlsanity.com/paper/2507.04984/) (2025) - [Textured 3D Regenerative Morphing with 3D Diffusion Prior](https://mlsanity.com/paper/2502.14316/) (2025) - [Interp3D: Correspondence-aware Interpolation for Generative Textured 3D Morphing](https://mlsanity.com/paper/2601.14103/) (2026) - [TwoSquared: 4D Generation from 2D Image Pairs](https://mlsanity.com/paper/2504.12825/) (2025) - [EventDiff: A Unified and Efficient Diffusion Model Framework for Event-based Video Frame Interpolation](https://mlsanity.com/paper/2505.08235/) (2025) - [Be Tangential to Manifold: Discovering Riemannian Metric for Diffusion Models](https://mlsanity.com/paper/2510.05509/) (2025) - [Time-Correlated Video Bridge Matching](https://mlsanity.com/paper/2510.12453/) (2025) - [AnchorSync: Global Consistency Optimization for Long Video Editing](https://mlsanity.com/paper/2508.14609/) (2025) - [Can Video Diffusion Models Predict Past Frames? Bidirectional Cycle Consistency for Reversible Interpolation](https://mlsanity.com/paper/2604.01700/) (2026) - [Ego-InBetween: Generating Object State Transitions in Ego-Centric Videos](https://mlsanity.com/paper/2604.17749/) (2026) - [Bridging Event Streams and DiT: Event-Guided Video Frame Interpolation](https://mlsanity.com/paper/2608.10479/) (2026) ## Abstract (arXiv) We study the problem of generating intermediate images from image pairs with large motion while maintaining semantic consistency. Due to the large motion, the intermediate semantic information may be absent in input images. Existing methods either limit to small motion or focus on topologically similar objects, leading to artifacts and inconsistency in the interpolation results. To overcome this challenge, we delve into pre-trained image diffusion models for their capabilities in semantic cognition and representations, ensuring consistent expression of the absent intermediate semantic representations with the input. To this end, we propose DreamMover, a novel image interpolation framework with three main components: 1) A natural flow estimator based on the diffusion model that can implicitly reason about the semantic correspondence between two images. 2) To avoid the loss of detailed information during fusion, our key insight is to fuse information in two parts, high-level space and low-level space. 3) To enhance the consistency between the generated images and input, we propose the self-attention concatenation and replacement approach. Lastly, we present a challenging benchmark dataset InterpBench to evaluate the semantic consistency of generated results. Extensive experiments demonstrate the effectiveness of our method. Our project is available at https://dreamm0ver.github.io .