mlsanityData to 14 Oct 2026

VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing

arXiv 2502.17258 ↗First posted 24 Feb 2025cs.CVcited by 50

On the map: Diffusion models → Video generation

Abstract (arXiv)

Recent advancements in diffusion models have significantly improved video generation and editing capabilities. However, multi-grained video editing, which encompasses class-level, instance-level, and part-level modifications, remains a formidable challenge. The major difficulties in multi-grained editing include semantic misalignment of text-to-region control and feature coupling within the diffusion model. To address these difficulties, we present VideoGrain, a zero-shot approach that modulates space-time (cross- and self-) attention mechanisms to achieve fine-grained control over video content. We enhance text-to-region control by amplifying each local prompt's attention to its corresponding spatial-disentangled region while minimizing interactions with irrelevant areas in cross-attention. Additionally, we improve feature separation by increasing intra-region awareness and reducing inter-region interference in self-attention. Extensive experiments demonstrate our method achieves state-of-the-art performance in real-world scenarios. Our code, data, and demos are available at https://knightyxp.github.io/VideoGrain_project_page/

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 46 of its 50 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

46 of 50 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  2. 2024FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention29 Jul
  3. 2024ReVideo: Remake a Video with Motion and Content Control22 May
  4. 2024EVA: Zero-shot Accurate Attributes and Multi-Object Video Editing24 Mar
  5. 2023VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence4 Dec
  6. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing9 Oct
  7. 2023Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models2 Oct
  8. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editing18 Aug
  9. 2023ModelScope Text-to-Video Technical Report12 Aug
  10. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  11. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation13 Jun
  12. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
3 more
  1. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  2. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  3. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec

Other references · 31

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2020Denoising Diffusion Implicit Modelscited by 9,781
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  4. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  5. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  6. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  7. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  8. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
  9. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  10. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  11. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  12. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
19 more
  1. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 647
  2. 2023Zero-shot Image-to-Image Translationcited by 505
  3. 2023Emergent Correspondence from Image Diffusioncited by 446
  4. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 442
  5. 2022Blended Latent Diffusioncited by 432
  6. 2022MAGVIT: Masked Generative Video Transformercited by 346
  7. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 314
  8. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 314
  9. 2023Segment and Track Anythingcited by 193
  10. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 174
  11. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 164
  12. 2023Dense Text-to-Image Generation with Attention Modulationcited by 163
  13. 2021Layered Neural Atlases for Consistent Video Editingcited by 137
  14. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  15. 2023CoDeF: Content Deformation Fields for Temporally Consistent Video Processingcited by 108
  16. 2023Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editingcited by 76
  17. 2023MagicStick: Controllable Video Editing via Control Handle Transformationscited by 62
  18. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 55
  19. 2024DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrievalcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 25; 2026: 25.

50 papers.

Sort:

By area:

  1. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  2. 2025Controllable Video Generation: A Surveycited by 98
  3. 2025Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuningcited by 81
  4. 2025Scaling Instruction-Based Video Editing with a High-Quality Synthetic Datasetcited by 79
  5. 2025EEdit: Rethinking the Spatial and Temporal Redundancy for Efficient Image Editingcited by 45
  6. 2025T2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generationcited by 19
  7. 2026Omni-Video 2: Scaling MLLM-Conditioned Diffusion for Unified Video Generation and Editingcited by 19
  8. 2025FiVE: A Fine-grained Video Editing Benchmark for Evaluating Emerging Diffusion and Rectified Flow Modelscited by 16
  9. 2026Past- and Future-Informed KV Cache Policy with Salience Estimation in Autoregressive Video Diffusioncited by 16
  10. 2026Bernini: Latent Semantic Planning for Video Diffusioncited by 12
  11. 2025FIA-Edit: Frequency-Interactive Attention for Efficient and High-Fidelity Inversion-Free Text-Guided Image Editingcited by 8
  12. 2025VideoCoF: Unified Video Editing with Temporal Reasonercited by 8
38 more
  1. 2026Context Unrolling in Omni Modelscited by 8
  2. 2025Towards Reason-Informed Video Editing in Unified Models with Self-Reflective Learningcited by 7
  3. 2025Causally Steered Diffusion for Automated Video Counterfactual Generationcited by 6
  4. 2025ASTRA: Let Arbitrary Subjects Transform in Video Editingcited by 6
  5. 2025FreeViS: Training-free Video Stylization with Inconsistent Referencescited by 6
  6. 2025Attention in Diffusion Model: A Surveycited by 4
  7. 2025DFVEdit: Conditional Delta Flow Vector for Zero-shot Video Editingcited by 4
  8. 2026STMI: Segmentation-Guided Token Modulation with Cross-Modal Hypergraph Interaction for Multi-Modal Object Re-Identificationcited by 3
  9. 2026Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoEcited by 3
  10. 2026Smart-Insertion-V: Photorealistic Video Insertion via a Closed-Loop Feedback Dual-Stream Frameworkcited by 3
  11. 2026ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationshipscited by 3
  12. 2026ElasticTTT: Prior-Preserving Test-Time Tuning for Video Editingcited by 3
  13. 2025Causal-Entity Reflected Egocentric Traffic Accident Video Synthesiscited by 2
  14. 2025Vectorized Video Representation with Easy Editing via Hierarchical Spatio-Temporally Consistent Proxy Embeddingcited by 2
  15. 2025FAME: Fairness-aware Attention-modulated Video Editingcited by 2
  16. 2025AlignVid: Training-Free Attention Scaling for Semantic Fidelity in Text-Guided Image-to-Video Generationcited by 2
  17. 2026ViFeEdit: A Video-Free Tuner of Your Video Diffusion Transformercited by 2
  18. 2026StreamEdit: Training-Free Video Editing via Few-Step Streaming Video Generationcited by 2
  19. 2025Low-Cost Test-Time Adaptation for Robust Video Editingcited by 1
  20. 2025Edit-Your-Interest: Efficient Video Editing via Feature Most-Similar Propagationcited by 1
  21. 2026RFDM: Residual Flow Diffusion Model for Efficient Causal Video Editingcited by 1
  22. 2026InsEdit: Towards Instruction-based Visual Editing via Data-Efficient Video Diffusion Models Adaptationcited by 1
  23. 2026FlowAnchor: Stabilizing the Editing Signal for Inversion-Free Video Editingcited by 1
  24. 2025STR-Match: Matching SpatioTemporal Relevance Score for Training-Free Video Editingcited by 0
  25. 2025HairShifter: Consistent and High-Fidelity Video Hair Transfer via Anchor-Guided Animationcited by 0
  26. 2025DreamSwapV: Mask-guided Subject Swapping for Any Customized Video Editingcited by 0
  27. 2025VALA: Learning Latent Anchors for Training-Free and Temporally Consistentcited by 0
  28. 2026Beyond Rigid: Benchmarking Non-Rigid Video Editingcited by 0
  29. 2026MotionGrounder: Grounded Multi-Object Motion Transfer via Diffusion Transformercited by 0
  30. 2026When Numbers Speak: Aligning Textual Numerals and Visual Instances in Text-to-Video Diffusion Modelscited by 0
  31. 2026Beyond Consistency: Preserving Temporal Structure in Zero-Shot Video Editingcited by 0
  32. 2026IDAG-Edit: Multi-Object Video Editing via Instance-Decoupled Attention and Guidancecited by 0
  33. 2026Encoder-Decoder Manifold Alignment for Idempotent Generationcited by 0
  34. 2026From Dense Prediction to Visual Editing: Structured Supervision for Unified Image and Video Creationcited by 0
  35. 2026One Editor, Many Edits: A Unified Training-Free Framework for Diverse Video Editingcited by 0
  36. 2026MDN-Control: Mask-Depth-Noise Guided Region Control for Multi-Subject Video Editingcited by 0
  37. 2026Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomycited by 0
  38. 2026Reimagine Video Dynamicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Investigating the Effectiveness of Cross-Attention to Unlock Zero-Shot Editing of Text-to-Video Diffusion Models
  2. 2025Zero-Shot Video Translation and Editing with Frame Spatial-Temporal Correspondence
  3. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation
  4. 2025Understanding Attention Mechanism in Video Diffusion Models
  5. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models
  6. 2023InFusion: Inject and Attention Fusion for Multi Concept Zero-Shot Text-based Video Editing
  7. 2025FramePainter: Endowing Interactive Image Editing with Video Diffusion Priors
  8. 2024Object-Centric Diffusion for Efficient Video Editing