mlsanityData to 7 Oct 2026

FateZero: Fusing Attentions for Zero-shot Text-based Video Editing

arXiv 2303.09535 ↗First posted 16 Mar 2023cs.CVcited by 462

On the map: Diffusion models → Video generation

Abstract (arXiv)

The diffusion-based generative models have achieved remarkable success in text-based image generation. However, since it contains enormous randomness in generation progress, it is still challenging to apply such models for real-world visual content editing, especially in videos. In this paper, we propose FateZero, a zero-shot text-based editing method on real-world videos without per-prompt training or use-specific mask. To edit videos consistently, we propose several techniques based on the pre-trained models. Firstly, in contrast to the straightforward DDIM inversion technique, our approach captures intermediate attention maps during inversion, which effectively retain both structural and motion information. These maps are directly fused in the editing process rather than generated during denoising. To further minimize semantic leakage of the source video, we then fuse self-attentions with a blending mask obtained by cross-attention features from the source prompt. Furthermore, we have implemented a reform of the self-attention mechanism in denoising UNet by introducing spatial-temporal attention to ensure frame consistency. Yet succinct, our method is the first one to show the ability of zero-shot text-driven video style and local attribute editing from the trained text-to-image model. We also have a better zero-shot shape-aware editing ability based on the text-to-video model. Extensive experiments demonstrate our superior temporal consistency and editing capability than previous works.

Key facts. Cited by 462 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 44 of its 60 references that are in the map. First posted on arXiv 16 Mar 2023.

Builds on

44 of 60 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  2. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb
  3. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  4. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  6. 2022Video Diffusion Models7 Apr

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  11. 2022Generative Adversarial Networkscited by 5,603
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
26 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  7. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  10. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  11. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  12. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  13. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  14. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  15. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  16. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  17. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  18. 2023Zero-shot Image-to-Image Translationcited by 503
  19. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  20. 2022Blended Latent Diffusioncited by 430
  21. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  22. 2018Learning Blind Video Temporal Consistencycited by 247
  23. 2021Layered Neural Atlases for Consistent Video Editingcited by 137
  24. 2020Blind Video Temporal Consistency via Deep Video Priorcited by 76
  25. 2023Shape-aware Text-driven Layered Video Editingcited by 54
  26. 2022Deep Video Prior for Video Consistency and Propagationcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 107; 2024: 146; 2025: 131; 2026: 78.

462 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  3. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolutioncited by 486
  4. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  5. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  6. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  7. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  8. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  9. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
  10. 2024FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Modelscited by 214
  11. 2023A Survey on Video Diffusion Modelscited by 203
  12. 2024Taming Rectified Flow for Inversion and Editingcited by 200
38 more
  1. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  2. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 172
  3. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  4. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 165
  5. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 151
  6. 2023Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Codecited by 145
  7. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 141
  8. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Modelscited by 135
  9. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 135
  10. 2023Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolutioncited by 131
  11. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  12. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  13. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  14. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  15. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 124
  16. 2024Training-Free Consistent Text-to-Image Generationcited by 123
  17. 2023RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Modelscited by 113
  18. 2023VidToMe: Video Token Merging for Zero-Shot Video Editingcited by 111
  19. 2023CoDeF: Content Deformation Fields for Temporally Consistent Video Processingcited by 108
  20. 2024Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidancecited by 108
  21. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105
  22. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  23. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 100
  24. 2024DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editingcited by 92
  25. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generationcited by 90
  26. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  27. 2023Improving Tuning-Free Real Image Editing with Proximal Guidancecited by 87
  28. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  29. 2023AVID: Any-Length Video Inpainting with Diffusion Modelcited by 81
  30. 2025Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuningcited by 81
  31. 2023CCEdit: Creative and Controllable Video Editing via Diffusion Modelscited by 80
  32. 2023GenTron: Diffusion Transformers for Image and Video Generationcited by 80
  33. 2023Consistent Video-to-Video Transfer Using Synthetic Datasetcited by 79
  34. 2023FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesiscited by 79
  35. 2024CustomVideo: Customizing Text-to-Video Generation with Multiple Subjectscited by 76
  36. 2025Scaling Instruction-Based Video Editing with a High-Quality Synthetic Datasetcited by 76
  37. 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
  38. 2024DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generationcited by 73

50 most cited of 462 All 462 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Investigating the Effectiveness of Cross-Attention to Unlock Zero-Shot Editing of Text-to-Video Diffusion Models
  2. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators
  3. 2025FluencyVE: Marrying Temporal-Aware Mamba with Bypass Attention for Video Editing
  4. 2024VideoDirector: Precise Video Editing via Text-to-Video Models
  5. 2023Pix2Video: Video Editing using Image Diffusion
  6. 2024Temporally Consistent Object Editing in Videos using Extended Attention
  7. 2024Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generation
  8. 2023FlowZero: Zero-Shot Text-to-Video Synthesis with LLM-Driven Dynamic Scene Syntax