mlsanityData to 7 Oct 2026

FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing

arXiv 2310.05922 ↗First posted 9 Oct 2023cs.CVcited by 165

On the map: Diffusion models → Video generation

Abstract (arXiv)

Text-to-video editing aims to edit the visual appearance of a source video conditional on textual prompts. A major challenge in this task is to ensure that all frames in the edited video are visually consistent. Most recent works apply advanced text-to-image diffusion models to this task by inflating 2D spatial attention in the U-Net into spatio-temporal attention. Although temporal context can be added through spatio-temporal attention, it may introduce some irrelevant information for each patch and therefore cause inconsistency in the edited video. In this paper, for the first time, we introduce optical flow into the attention module in the diffusion model's U-Net to address the inconsistency issue for text-to-video editing. Our method, FLATTEN, enforces the patches on the same flow path across different frames to attend to each other in the attention module, thus improving the visual consistency in the edited videos. Additionally, our method is training-free and can be seamlessly integrated into any diffusion-based text-to-video editing methods and improve their visual consistency. Experiment results on existing text-to-video editing benchmarks show that our proposed method achieves the new state-of-the-art performance. In particular, our method excels in maintaining the visual consistency in the edited videos.

Key facts. Cited by 165 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 51 of its 59 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

51 of 59 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
  2. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  3. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesis27 Apr
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  5. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  6. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  7. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  8. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  9. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  10. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  11. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  12. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
1 more
  1. 2022Video Diffusion Models7 Apr

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
26 more
  1. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  2. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  5. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  6. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  7. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  8. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  9. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  10. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  11. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  12. 2023Zero-shot Image-to-Image Translationcited by 503
  13. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  14. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  15. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  16. 2022MAGVIT: Masked Generative Video Transformercited by 341
  17. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  18. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  19. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  20. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  21. 2018Learning Blind Video Temporal Consistencycited by 247
  22. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  23. 2021Learning to Estimate Hidden Motions with Global Motion Aggregationcited by 223
  24. 2023Generative Image Dynamicscited by 93
  25. 2021CCVS: Context-aware Controllable Video Synthesiscited by 68
  26. 2022SSGVS: Semantic Scene Graph-to-Video Synthesiscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 46; 2025: 67; 2026: 41.

165 papers.

Sort:

By area:

  1. 2025VACE: All-in-One Video Creation and Editingcited by 467
  2. 2023A Survey on Video Diffusion Modelscited by 203
  3. 2024Taming Rectified Flow for Inversion and Editingcited by 200
  4. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 151
  5. 2023Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolutioncited by 131
  6. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 130
  7. 2023RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Modelscited by 113
  8. 2025VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Modelscited by 112
  9. 2023GenTron: Diffusion Transformers for Image and Video Generationcited by 80
  10. 2023FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesiscited by 79
  11. 2024ReVideo: Remake a Video with Motion and Content Controlcited by 74
  12. 2023MagicStick: Controllable Video Editing via Control Handle Transformationscited by 62
38 more
  1. 2025InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Constructioncited by 62
  2. 2025Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialistscited by 61
  3. 2025Follow-Your-Creation: Empowering 4D Creation through Video Inpaintingcited by 61
  4. 2024COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editingcited by 59
  5. 2024I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Modelscited by 55
  6. 2025VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editingcited by 50
  7. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49
  8. 2024FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translationcited by 48
  9. 2024UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editingcited by 47
  10. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  11. 2025MiniMax-Remover: Taming Bad Noise Helps Video Object Removalcited by 42
  12. 2024Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slicescited by 40
  13. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
  14. 2024VideoDirector: Precise Video Editing via Text-to-Video Modelscited by 36
  15. 2024Generative Video Propagationcited by 36
  16. 2025In-Context Learning with Unpaired Clips for Instruction-based Video Editingcited by 34
  17. 2026Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidancecited by 34
  18. 2024DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Modelscited by 33
  19. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  20. 2025InstructX: Towards Unified Visual Editing with MLLM Guidancecited by 31
  21. 2024MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusioncited by 29
  22. 2024DreamMotion: Space-Time Self-Similar Score Distillation for Zero-Shot Video Editingcited by 28
  23. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  24. 2025LoRA-Edit: Controllable First-Frame-Guided Video Editing via Mask-Aware LoRA Fine-Tuningcited by 27
  25. 2025IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessmentcited by 27
  26. 2024StableV2V: Stablizing Shape Consistency in Video-to-Video Editingcited by 25
  27. 2024VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editingcited by 24
  28. 2025MatAnyone: Stable Video Matting with Consistent Memory Propagationcited by 24
  29. 2025FlowDirector: Training-Free Flow Steering for Precise Text-to-Video Editingcited by 24
  30. 2025VideoGen-Eval: Agent-based System for Video Generation Evaluationcited by 23
  31. 2023Drag-A-Video: Non-rigid Video Editing with Point-based Interactioncited by 22
  32. 2025Region-Constraint In-Context Generation for Instructional Video Editingcited by 22
  33. 2025FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generationcited by 21
  34. 2025Generative Video Motion Editing with 3D Point Trackscited by 21
  35. 2025EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidancecited by 20
  36. 2024EVA: Zero-shot Accurate Attributes and Multi-Object Video Editingcited by 17
  37. 2024DIVE: Taming DINO for Subject-Driven Video Editingcited by 17
  38. 2024UniVST: A Unified Framework for Training-free Localized Video Style Transfercited by 16

50 most cited of 165 All 165 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Consistent Video Editing with Text-to-Image Diffusion Models
  2. 2023Edit Temporal-Consistent Videos with Image Diffusion Model
  3. 2024Temporally Consistent Object Editing in Videos using Extended Attention
  4. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editing
  5. 2024Investigating the Effectiveness of Cross-Attention to Unlock Zero-Shot Editing of Text-to-Video Diffusion Models
  6. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models
  7. 2023InFusion: Inject and Attention Fusion for Multi Concept Zero-Shot Text-based Video Editing
  8. 2025FADE: Frequency-Aware Diffusion Model Factorization for Video Editing