# FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing

arXiv: https://arxiv.org/abs/2310.05922 · HTML: https://mlsanity.com/paper/2310.05922/

## Key facts

- Cited by 167 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 51 of its 59 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 51)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)

## Built on by (most cited first, 30 of 167)

- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [A Survey on Video Diffusion Models](https://mlsanity.com/paper/2310.10647/) (2023)
- [Taming Rectified Flow for Inversion and Editing](https://mlsanity.com/paper/2411.04746/) (2024)
- [AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks](https://mlsanity.com/paper/2403.14468/) (2024)
- [Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution](https://mlsanity.com/paper/2312.06640/) (2023)
- [ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation](https://mlsanity.com/paper/2402.04324/) (2024)
- [RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models](https://mlsanity.com/paper/2312.04524/) (2023)
- [VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models](https://mlsanity.com/paper/2502.02492/) (2025)
- [GenTron: Diffusion Transformers for Image and Video Generation](https://mlsanity.com/paper/2312.04557/) (2023)
- [FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis](https://mlsanity.com/paper/2312.17681/) (2023)
- [ReVideo: Remake a Video with Motion and Content Control](https://mlsanity.com/paper/2405.13865/) (2024)
- [MagicStick: Controllable Video Editing via Control Handle Transformations](https://mlsanity.com/paper/2312.03047/) (2023)
- [InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Construction](https://mlsanity.com/paper/2503.20287/) (2025)
- [Follow-Your-Creation: Empowering 4D Creation through Video Inpainting](https://mlsanity.com/paper/2506.04590/) (2025)
- [Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists](https://mlsanity.com/paper/2502.06734/) (2025)
- [COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing](https://mlsanity.com/paper/2406.08850/) (2024)
- [I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models](https://mlsanity.com/paper/2405.16537/) (2024)
- [VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing](https://mlsanity.com/paper/2502.17258/) (2025)
- [FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation](https://mlsanity.com/paper/2403.12962/) (2024)
- [MarDini: Masked Autoregressive Diffusion for Video Generation at Scale](https://mlsanity.com/paper/2410.20280/) (2024)
- [UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing](https://mlsanity.com/paper/2402.13185/) (2024)
- [MiniMax-Remover: Taming Bad Noise Helps Video Object Removal](https://mlsanity.com/paper/2505.24873/) (2025)
- [Diffusion Model-Based Video Editing: A Survey](https://mlsanity.com/paper/2407.07111/) (2024)
- [Efficient Diffusion Models: A Comprehensive Survey from Principles to Practices](https://mlsanity.com/paper/2410.11795/) (2024)
- [Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slices](https://mlsanity.com/paper/2405.12211/) (2024)
- [VideoDirector: Precise Video Editing via Text-to-Video Models](https://mlsanity.com/paper/2411.17592/) (2024)
- [Generative Video Propagation](https://mlsanity.com/paper/2412.19761/) (2024)
- [Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance](https://mlsanity.com/paper/2603.02175/) (2026)
- [In-Context Learning with Unpaired Clips for Instruction-based Video Editing](https://mlsanity.com/paper/2510.14648/) (2025)
- [DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models](https://mlsanity.com/paper/2407.01519/) (2024)

## Abstract (arXiv)

Text-to-video editing aims to edit the visual appearance of a source video conditional on textual prompts. A major challenge in this task is to ensure that all frames in the edited video are visually consistent. Most recent works apply advanced text-to-image diffusion models to this task by inflating 2D spatial attention in the U-Net into spatio-temporal attention. Although temporal context can be added through spatio-temporal attention, it may introduce some irrelevant information for each patch and therefore cause inconsistency in the edited video. In this paper, for the first time, we introduce optical flow into the attention module in the diffusion model's U-Net to address the inconsistency issue for text-to-video editing. Our method, FLATTEN, enforces the patches on the same flow path across different frames to attend to each other in the attention module, thus improving the visual consistency in the edited videos. Additionally, our method is training-free and can be seamlessly integrated into any diffusion-based text-to-video editing methods and improve their visual consistency. Experiment results on existing text-to-video editing benchmarks show that our proposed method achieves the new state-of-the-art performance. In particular, our method excels in maintaining the visual consistency in the edited videos.