# Pathways on the Image Manifold: Image Editing via Video Generation

arXiv: https://arxiv.org/abs/2411.16819 · HTML: https://mlsanity.com/paper/2411.16819/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 37 of its 50 references that are in the map.
- First posted on arXiv 25 Nov 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [AnyDoor: Zero-shot Object-level Image Customization](https://mlsanity.com/paper/2307.09481/) (2023)
- [Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers](https://mlsanity.com/paper/2402.19479/) (2024)
- [Generating Videos with Scene Dynamics](https://mlsanity.com/paper/1609.02612/) (2016)
- [An Edit Friendly DDPM Noise Space: Inversion and Manipulations](https://mlsanity.com/paper/2304.06140/) (2023)
- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [Text-To-4D Dynamic Scene Generation](https://mlsanity.com/paper/2301.11280/) (2023)
- [PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation](https://mlsanity.com/paper/2404.13026/) (2024)
- [An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM](https://mlsanity.com/paper/2403.18406/) (2024)

## Built on by (most cited first, 27 of 27)

- [Video models are zero-shot learners and reasoners](https://mlsanity.com/paper/2509.20328/) (2025)
- [Motion Prompting: Controlling Video Generation with Motion Trajectories](https://mlsanity.com/paper/2412.02700/) (2024)
- [ChronoEdit: Towards Temporal Reasoning for Image Editing and World Simulation](https://mlsanity.com/paper/2510.04290/) (2025)
- [ByteMorph: Benchmarking Instruction-Guided Image Editing with Non-Rigid Motions](https://mlsanity.com/paper/2506.03107/) (2025)
- [PICABench: How Far Are We from Physically Realistic Image Editing?](https://mlsanity.com/paper/2510.17681/) (2025)
- [FramePainter: Endowing Interactive Image Editing with Video Diffusion Priors](https://mlsanity.com/paper/2501.08225/) (2025)
- [Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising](https://mlsanity.com/paper/2511.08633/) (2025)
- [Group Editing: Edit Multiple Images in One Go](https://mlsanity.com/paper/2603.22883/) (2026)
- [Image Editing As Programs with Diffusion Models](https://mlsanity.com/paper/2506.04158/) (2025)
- [Olaf-World: Orienting Latent Actions for Video World Modeling](https://mlsanity.com/paper/2602.10104/) (2026)
- [UniEdit-I: Training-free Image Editing for Unified VLM via Iterative Understanding, Editing and Verifying](https://mlsanity.com/paper/2508.03142/) (2025)
- [iMontage: Unified, Versatile, Highly Dynamic Many-to-many Image Generation](https://mlsanity.com/paper/2511.20635/) (2025)
- [Are Image-to-Video Models Good Zero-Shot Image Editors?](https://mlsanity.com/paper/2511.19435/) (2025)
- [Beyond the Last Frame: Process-aware Evaluation for Generative Video Reasoning](https://mlsanity.com/paper/2512.24952/) (2025)
- [From Statics to Dynamics: Physics-Aware Image Editing with Latent Transition Priors](https://mlsanity.com/paper/2602.21778/) (2026)
- [PhyEditBench: A Real-World Multi-Stage Benchmark for Physics-Aware Image Editing](https://mlsanity.com/paper/2606.26551/) (2026)
- [Neural Descriptors: Self-Supervised Learning of Robust Local Surface Descriptors Using Polynomial Patches](https://mlsanity.com/paper/2503.03907/) (2025)
- [AnchorEdit: Maintaining Temporal Consistency in Multi-turn Image Editing via Causal Memory](https://mlsanity.com/paper/2606.11751/) (2026)
- [DreamingComics: A Story Visualization Pipeline via Subject and Layout Customized Generation using Video Models](https://mlsanity.com/paper/2512.01686/) (2025)
- [PhyEdit: Towards Real-World Object Manipulation via Physically-Grounded Image Editing](https://mlsanity.com/paper/2604.07230/) (2026)
- [ScrollScape: Unlocking 32K Image Generation With Video Diffusion Priors](https://mlsanity.com/paper/2603.24270/) (2026)
- [UniGeo: Unifying Geometric Guidance for Camera-Controllable Image Editing via Video Models](https://mlsanity.com/paper/2604.17565/) (2026)
- [GeoEdit: Local Frames for Fast, Training-Free On-Manifold Editing in Diffusion Models](https://mlsanity.com/paper/2604.24238/) (2026)
- [PhysEdit: Physically-Consistent Region-Aware Image Editing via Adaptive Spatio-Temporal Reasoning](https://mlsanity.com/paper/2605.00707/) (2026)
- [Bootstrap Your Generator: Unpaired Visual Editing with Flow Matching](https://mlsanity.com/paper/2606.03911/) (2026)
- [Token-to-Token Alignment of Text Embeddings for Semantic Blending](https://mlsanity.com/paper/2606.24021/) (2026)
- [Bridging Video Understanding and Generation in a Unified Framework](https://mlsanity.com/paper/2606.31326/) (2026)

## Abstract (arXiv)

Recent advances in image editing, driven by image diffusion models, have shown remarkable progress. However, significant challenges remain, as these models often struggle to follow complex edit instructions accurately and frequently compromise fidelity by altering key elements of the original image. Simultaneously, video generation has made remarkable strides, with models that effectively function as consistent and continuous world simulators. In this paper, we propose merging these two fields by utilizing image-to-video models for image editing. We reformulate image editing as a temporal process, using pretrained video models to create smooth transitions from the original image to the desired edit. This approach traverses the image manifold continuously, ensuring consistent edits while preserving the original image's key aspects. Our approach achieves state-of-the-art results on text-based image editing, demonstrating significant improvements in both edit accuracy and image preservation. Visit our project page at https://rotsteinnoam.github.io/Frame2Frame.