mlsanityData to 14 Oct 2026

Pathways on the Image Manifold: Image Editing via Video Generation

arXiv 2411.16819 ↗First posted 25 Nov 2024cs.CV cs.AI cs.LGcited by 27

On the map: Diffusion models → Video generation

Abstract (arXiv)

Recent advances in image editing, driven by image diffusion models, have shown remarkable progress. However, significant challenges remain, as these models often struggle to follow complex edit instructions accurately and frequently compromise fidelity by altering key elements of the original image. Simultaneously, video generation has made remarkable strides, with models that effectively function as consistent and continuous world simulators. In this paper, we propose merging these two fields by utilizing image-to-video models for image editing. We reformulate image editing as a temporal process, using pretrained video models to create smooth transitions from the original image to the desired edit. This approach traverses the image manifold continuously, ensuring consistent edits while preserving the original image's key aspects. Our approach achieves state-of-the-art results on text-based image editing, demonstrating significant improvements in both edit accuracy and image preservation. Visit our project page at https://rotsteinnoam.github.io/Frame2Frame.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 37 of its 50 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

37 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  3. 2023AnyDoor: Zero-shot Object-level Image Customization18 Jul
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  5. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  9. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  10. 2018World Modelscited by 1,513
  11. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  12. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
20 more
  1. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  2. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  3. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  4. 2023Zero-shot Image-to-Image Translationcited by 505
  5. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  6. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  7. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  8. 2016Generating Videos with Scene Dynamicscited by 376
  9. 2023An Edit Friendly DDPM Noise Space: Inversion and Manipulationscited by 244
  10. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  11. 2023Text-To-4D Dynamic Scene Generationcited by 205
  12. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 191
  13. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  14. 2023ViVid-1-to-3: Novel View Synthesis with Video Diffusion Modelscited by 58
  15. 2023Readout Guidance: Learning Control from Diffusion Featurescited by 52
  16. 2023High-Fidelity and Freely Controllable Talking Head Video Generationcited by 42
  17. 2024Paint by Inpaint: Learning to Add Image Objects by Removing Them Firstcited by 39
  18. 2024Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videoscited by 38
  19. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captionscited by 33
  20. 2024Question Aware Vision Transformer for Multimodal Reasoningcited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 13.

27 papers.

Sort:

By area:

  1. 2025Video models are zero-shot learners and reasonerscited by 220
  2. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 178
  3. 2025ChronoEdit: Towards Temporal Reasoning for Image Editing and World Simulationcited by 29
  4. 2025ByteMorph: Benchmarking Instruction-Guided Image Editing with Non-Rigid Motionscited by 19
  5. 2025PICABench: How Far Are We from Physically Realistic Image Editing?cited by 16
  6. 2025FramePainter: Endowing Interactive Image Editing with Video Diffusion Priorscited by 13
  7. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoisingcited by 13
  8. 2026Group Editing: Edit Multiple Images in One Gocited by 11
  9. 2025Image Editing As Programs with Diffusion Modelscited by 10
  10. 2026Olaf-World: Orienting Latent Actions for Video World Modelingcited by 10
  11. 2025UniEdit-I: Training-free Image Editing for Unified VLM via Iterative Understanding, Editing and Verifyingcited by 7
  12. 2025iMontage: Unified, Versatile, Highly Dynamic Many-to-many Image Generationcited by 6
15 more
  1. 2025Are Image-to-Video Models Good Zero-Shot Image Editors?cited by 4
  2. 2025Beyond the Last Frame: Process-aware Evaluation for Generative Video Reasoningcited by 4
  3. 2026From Statics to Dynamics: Physics-Aware Image Editing with Latent Transition Priorscited by 3
  4. 2026PhyEditBench: A Real-World Multi-Stage Benchmark for Physics-Aware Image Editingcited by 3
  5. 2025Neural Descriptors: Self-Supervised Learning of Robust Local Surface Descriptors Using Polynomial Patchescited by 2
  6. 2026AnchorEdit: Maintaining Temporal Consistency in Multi-turn Image Editing via Causal Memorycited by 2
  7. 2025DreamingComics: A Story Visualization Pipeline via Subject and Layout Customized Generation using Video Modelscited by 1
  8. 2026PhyEdit: Towards Real-World Object Manipulation via Physically-Grounded Image Editingcited by 1
  9. 2026ScrollScape: Unlocking 32K Image Generation With Video Diffusion Priorscited by 0
  10. 2026UniGeo: Unifying Geometric Guidance for Camera-Controllable Image Editing via Video Modelscited by 0
  11. 2026GeoEdit: Local Frames for Fast, Training-Free On-Manifold Editing in Diffusion Modelscited by 0
  12. 2026PhysEdit: Physically-Consistent Region-Aware Image Editing via Adaptive Spatio-Temporal Reasoningcited by 0
  13. 2026Bootstrap Your Generator: Unpaired Visual Editing with Flow Matchingcited by 0
  14. 2026Token-to-Token Alignment of Text Embeddings for Semantic Blendingcited by 0
  15. 2026Bridging Video Understanding and Generation in a Unified Frameworkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Editing on the Generative Manifold: A Theoretical and Empirical Study of General Diffusion-Based Image Editing Trade-offs
  2. 2025Image Editing with Diffusion Models: A Survey
  3. 2023DiffusionAtlas: High-Fidelity Consistent Diffusion Video Editing
  4. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editing
  5. 2023Pix2Video: Video Editing using Image Diffusion
  6. 2023VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing
  7. 2024EditWorld: Simulating World Dynamics for Instruction-Following Image Editing
  8. 2026Transforming Image Editors into Video Editors