mlsanityData to 7 Oct 2026

Pix2Video: Video Editing using Image Diffusion

arXiv 2303.12688 ↗First posted 22 Mar 2023cs.CVcited by 318

On the map: Diffusion models → Video generation

Abstract (arXiv)

Image diffusion models, trained on massive image collections, have emerged as the most versatile image generator model in terms of quality and diversity. They support inverting real images and conditional (e.g., text) generation, making them attractive for high-quality image editing applications. We investigate how to use such pre-trained image models for text-guided video editing. The critical challenge is to achieve the target edits while still preserving the content of the source video. Our method works in two simple steps: first, we use a pre-trained structure-guided (e.g., depth) image diffusion model to perform text-guided edits on an anchor frame; then, in the key step, we progressively propagate the changes to the future frames via self-attention feature injection to adapt the core denoising step of the diffusion model. We then consolidate the changes by adjusting the latent code for the frame before continuing the process. Our approach is training-free and generalizes to a wide range of edits. We demonstrate the effectiveness of the approach by extensive experimentation and compare it against four different prior and parallel efforts (on ArXiv). We demonstrate that realistic text-guided video edits are possible, without any compute-intensive preprocessing or video-specific finetuning.

Key facts. Cited by 318 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 58 of its 68 references that are in the map. First posted on arXiv 22 Mar 2023.

Builds on

58 of 68 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Video Probabilistic Diffusion Models in Projected Latent Space15 Feb
  2. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  3. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb
  4. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct

Other references · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2017Mask R-CNNcited by 7,259
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
38 more
  1. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  2. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  4. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  6. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  7. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  8. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  9. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  10. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  11. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  12. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  13. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  14. 2022Diffusion Models in Vision: A Surveycited by 1,008
  15. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  16. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  17. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  18. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  19. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  20. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  21. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  22. 2018Video-to-Video Synthesiscited by 388
  23. 2022MAGVIT: Masked Generative Video Transformercited by 341
  24. 2022Diffusion Models already have a Semantic Latent Spacecited by 324
  25. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  26. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  27. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  28. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  29. 2019Few-shot Video-to-Video Synthesiscited by 196
  30. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  31. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  32. 2022Pretraining is All You Need for Image-to-Image Translationcited by 160
  33. 2021Layered Neural Atlases for Consistent Video Editingcited by 137
  34. 2022Generating Long Videos of Dynamic Scenescited by 113
  35. 2016Artistic style transfer for videoscited by 54
  36. 2020Layered Neural Rendering for Retiming People in Videocited by 51
  37. 2020Interactive Video Stylization Using Few-Shot Patch-Based Trainingcited by 43
  38. 2022Temporally Consistent Semantic Video Editingcited by 28

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 73; 2024: 112; 2025: 93; 2026: 40.

318 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  3. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  4. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  5. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  6. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  7. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  8. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  9. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
  10. 2023A Survey on Video Diffusion Modelscited by 203
  11. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  12. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
38 more
  1. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 165
  2. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  3. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 151
  4. 2023Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Modelscited by 148
  5. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  6. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Modelscited by 135
  7. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 135
  8. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 130
  9. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  10. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  11. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 124
  12. 2024Training-Free Consistent Text-to-Image Generationcited by 123
  13. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  14. 2023RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Modelscited by 113
  15. 2023VidToMe: Video Token Merging for Zero-Shot Video Editingcited by 111
  16. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  17. 2024DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editingcited by 92
  18. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  19. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  20. 2023VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Modelscited by 86
  21. 2024Animate-X: Universal Character Image Animation with Enhanced Motion Representationcited by 86
  22. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  23. 2023AVID: Any-Length Video Inpainting with Diffusion Modelcited by 81
  24. 2023CCEdit: Creative and Controllable Video Editing via Diffusion Modelscited by 80
  25. 2023Consistent Video-to-Video Transfer Using Synthetic Datasetcited by 79
  26. 2023FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesiscited by 79
  27. 2025Scaling Instruction-Based Video Editing with a High-Quality Synthetic Datasetcited by 76
  28. 2023MagicEdit: High-Fidelity and Temporally Coherent Video Editingcited by 75
  29. 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
  30. 2024DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videoscited by 63
  31. 2025Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialistscited by 61
  32. 2025Follow-Your-Creation: Empowering 4D Creation through Video Inpaintingcited by 61
  33. 2025VideoVLA: Video Generators Can Be Generalizable Robot Manipulatorscited by 61
  34. 2023Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Modelscited by 58
  35. 2023MotionEditor: Editing Video Motion via Content-Aware Diffusioncited by 56
  36. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 56
  37. 2024Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Modelscited by 55
  38. 2023CVPR 2023 Text Guided Video Editing Competitioncited by 54

50 most cited of 318 All 318 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion Model-Based Image Editing: A Survey
  2. 2023Zero-shot Image-to-Image Translation
  3. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editing
  4. 2025Contrastive Learning Guided Latent Diffusion Model for Image-to-Image Translation
  5. 2024I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models
  6. 2025FADE: Frequency-Aware Diffusion Model Factorization for Video Editing
  7. 2025Image Editing with Diffusion Models: A Survey
  8. 2023PRedItOR: Text Guided Image Editing with Diffusion Prior