mlsanityData to 7 Oct 2026

VidToMe: Video Token Merging for Zero-Shot Video Editing

arXiv 2312.10656 ↗First posted 17 Dec 2023cs.CVcited by 111

On the map: Diffusion models → Video generation

also in 1 more
Abstract (arXiv)

Diffusion models have made significant advances in generating high-quality images, but their application to video generation has remained challenging due to the complexity of temporal motion. Zero-shot video editing offers a solution by utilizing pre-trained image diffusion models to translate source videos into new ones. Nevertheless, existing methods struggle to maintain strict temporal consistency and efficient memory consumption. In this work, we propose a novel approach to enhance temporal consistency in generated videos by merging self-attention tokens across frames. By aligning and compressing temporally redundant tokens across frames, our method improves temporal coherence and reduces memory consumption in self-attention computations. The merging strategy matches and aligns tokens according to the temporal correspondence between frames, facilitating natural temporal consistency in generated video frames. To manage the complexity of video processing, we divide videos into chunks and develop intra-chunk local token merging and inter-chunk global token merging, ensuring both short-term video continuity and long-term content consistency. Our video editing approach seamlessly extends the advancements in image editing to video editing, rendering favorable results in temporal consistency over state-of-the-art methods.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 51 of its 58 references that are in the map. First posted on arXiv 17 Dec 2023.

Builds on

51 of 58 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation13 Jun
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  3. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models30 Mar
  4. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  5. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  6. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  7. 2023Video Probabilistic Diffusion Models in Projected Latent Space15 Feb
  8. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  9. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  10. 2022Video Diffusion Models7 Apr

Other references · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
29 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  6. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  7. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  8. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  9. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  10. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  11. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  12. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  13. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  14. 2022Diffusion Models in Vision: A Surveycited by 1,008
  15. 2022Token Merging: Your ViT But Fastercited by 995
  16. 2022Denoising Diffusion Restoration Modelscited by 951
  17. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  18. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  19. 2017Generative Adversarial Networks: An Overviewcited by 588
  20. 2023Zero-shot Image-to-Image Translationcited by 503
  21. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  22. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  23. 2017Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolationcited by 321
  24. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  25. 2023Token Merging for Fast Stable Diffusioncited by 220
  26. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  27. 2023CVPR 2023 Text Guided Video Editing Competitioncited by 54
  28. 2020Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applicationscited by 48
  29. 2020Video Generative Adversarial Networks: A Reviewcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 46; 2026: 37.

111 papers.

Sort:

By area:

  1. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  2. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 100
  3. 2025InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Constructioncited by 62
  4. 2025Hunyuan-GameCraft-2: Instruction-following Interactive Game World Modelcited by 58
  5. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  6. 2024UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editingcited by 47
  7. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  8. 2024Video Editing via Factorized Diffusion Distillationcited by 40
  9. 2024VideoDirector: Precise Video Editing via Text-to-Video Modelscited by 36
  10. 2024Generative Video Propagationcited by 36
  11. 2025In-Context Learning with Unpaired Clips for Instruction-based Video Editingcited by 34
  12. 2024DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Modelscited by 33
38 more
  1. 2024Object-Centric Diffusion for Efficient Video Editingcited by 31
  2. 2024Splatter a Video: Video Gaussian Representation for Versatile Processingcited by 31
  3. 2025LoRA-Edit: Controllable First-Frame-Guided Video Editing via Mask-Aware LoRA Fine-Tuningcited by 27
  4. 2025IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessmentcited by 27
  5. 2025Generative Physical AI in Vision: A Surveycited by 24
  6. 2024Video Token Merging for Long-form Video Understandingcited by 22
  7. 2025Region-Constraint In-Context Generation for Instructional Video Editingcited by 22
  8. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 20
  9. 2025Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Modelcited by 19
  10. 2026Omni-Video 2: Scaling MLLM-Conditioned Diffusion for Unified Video Generation and Editingcited by 18
  11. 2024DIVE: Taming DINO for Subject-Driven Video Editingcited by 17
  12. 2025VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimizationcited by 16
  13. 2024AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restorationcited by 15
  14. 2025FiVE: A Fine-grained Video Editing Benchmark for Evaluating Emerging Diffusion and Rectified Flow Modelscited by 15
  15. 2025On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devicescited by 14
  16. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  17. 2025Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoisingcited by 13
  18. 2024Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Drivingcited by 11
  19. 2024SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editingcited by 11
  20. 2025TC-Light: Temporally Coherent Generative Rendering for Realistic World Transfercited by 11
  21. 2026Bernini: Latent Semantic Planning for Video Diffusioncited by 11
  22. 2024Animate Your Motion: Turning Still Images into Dynamic Videoscited by 10
  23. 2024Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editingcited by 10
  24. 2024Leveraging Temporal Contextualization for Video Action Recognitioncited by 9
  25. 2024VeGaS: Video Gaussian Splattingcited by 9
  26. 2025VidSketch: Hand-drawn Sketch-Driven Video Generation with Diffusion Controlcited by 9
  27. 2025Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Accelerationcited by 9
  28. 2025AVI-Edit: Audio-sync Video Instance Editing with Granularity-Aware Mask Refinercited by 8
  29. 2026Coarse-to-Real: Generative Rendering for Populated Dynamic Scenescited by 8
  30. 2026Generative World Renderercited by 8
  31. 2026Context Unrolling in Omni Modelscited by 8
  32. 2024NaRCan: Natural Refined Canonical Image with Integration of Diffusion Prior for Video Editingcited by 7
  33. 2025VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Ratecited by 7
  34. 2025FreeViS: Training-free Video Stylization with Inconsistent Referencescited by 6
  35. 2026A Mechanistic View on Video Generation as World Models: State and Dynamicscited by 6
  36. 2026LongVideo-R1: Smart Navigation for Low-cost Long Video Understandingcited by 6
  37. 2024B-VLLM: A Vision Large Language Model with Balanced Spatio-Temporal Tokenscited by 5
  38. 2025A Similarity Paradigm Through Textual Regularization Without Forgettingcited by 5

50 most cited of 111 All 111 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Zero-Shot Video Translation via Token Warping
  2. 2023LatentWarp: Consistent Diffusion Latents for Zero-Shot Video-to-Video Translation
  3. 2023VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing
  4. 2025TokensGen: Harnessing Condensed Tokens for Long Video Generation
  5. 2026Temporal Aware Pruning for Efficient Diffusion-based Video Generation
  6. 2024COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing
  7. 2024FastVideoEdit: Leveraging Consistency Models for Efficient Text-to-Video Editing
  8. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editing