mlsanityData to 7 Oct 2026

Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation

arXiv 2306.07954 ↗First posted 13 Jun 2023cs.CVcited by 253

On the map: Diffusion models → Video generation

Abstract (arXiv)

Large text-to-image diffusion models have exhibited impressive proficiency in generating high-quality images. However, when applying these models to video domain, ensuring temporal consistency across video frames remains a formidable challenge. This paper proposes a novel zero-shot text-guided video-to-video translation framework to adapt image models to videos. The framework includes two parts: key frame translation and full video translation. The first part uses an adapted diffusion model to generate key frames, with hierarchical cross-frame constraints applied to enforce coherence in shapes, textures and colors. The second part propagates the key frames to other frames with temporal-aware patch matching and frame blending. Our framework achieves global style and local texture temporal consistency at a low cost (without re-training or optimization). The adaptation is compatible with existing image diffusion techniques, allowing our framework to take advantage of them, such as customizing a specific subject with LoRA, and introducing extra spatial guidance with ControlNet. Extensive experimental results demonstrate the effectiveness of our proposed framework over existing methods in rendering high-quality and temporally-coherent videos.

Key facts. Cited by 253 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 41 of its 48 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

41 of 48 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models30 Mar
  2. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  3. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  4. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  5. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb
  6. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  8. 2022Video Diffusion Models7 Apr

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
21 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  5. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  6. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  7. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  10. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  11. 2022Diffusion Models in Vision: A Surveycited by 1,008
  12. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  13. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  14. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  15. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  16. 2022Blended Latent Diffusioncited by 430
  17. 2021GMFlow: Learning Optical Flow via Global Matchingcited by 324
  18. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  19. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  20. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  21. 2023Edit-A-Video: Single Video Editing with Object-Aware Consistencycited by 65

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 51; 2024: 98; 2025: 69; 2026: 35.

253 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  3. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  4. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  5. 2023A Survey on Video Diffusion Modelscited by 203
  6. 2024Taming Rectified Flow for Inversion and Editingcited by 200
  7. 2023FreeU: Free Lunch in Diffusion U-Netcited by 195
  8. 2023Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfercited by 194
  9. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  10. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 151
  11. 2023Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolutioncited by 131
  12. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
38 more
  1. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 124
  2. 2023RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Modelscited by 113
  3. 2023VidToMe: Video Token Merging for Zero-Shot Video Editingcited by 111
  4. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  5. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 100
  6. 2025Controllable Video Generation: A Surveycited by 93
  7. 2023Animate-A-Story: Storytelling with Retrieval-Augmented Video Generationcited by 89
  8. 2023CCEdit: Creative and Controllable Video Editing via Diffusion Modelscited by 80
  9. 2023Consistent Video-to-Video Transfer Using Synthetic Datasetcited by 79
  10. 2023FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesiscited by 79
  11. 2023MagicEdit: High-Fidelity and Temporally Coherent Video Editingcited by 75
  12. 2023VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondencecited by 69
  13. 2024EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animationcited by 69
  14. 2024A Survey on LoRA of Large Language Modelscited by 67
  15. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  16. 2023The Chosen One: Consistent Characters in Text-to-Image Diffusion Modelscited by 62
  17. 2023MagicStick: Controllable Video Editing via Control Handle Transformationscited by 62
  18. 2024Visual Style Prompting with Swapping Self-Attentioncited by 62
  19. 2024Video Diffusion Models are Training-free Motion Interpreter and Controllercited by 60
  20. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generationcited by 59
  21. 2024COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editingcited by 59
  22. 2024ViewDiff: 3D-Consistent Image Generation with Text-to-Image Modelscited by 57
  23. 2024360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Modelcited by 56
  24. 2024Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Modelscited by 55
  25. 2024I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Modelscited by 55
  26. 2023CVPR 2023 Text Guided Video Editing Competitioncited by 54
  27. 2025VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editingcited by 50
  28. 2024FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translationcited by 48
  29. 2024UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editingcited by 47
  30. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  31. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  32. 2023DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatarscited by 42
  33. 2024Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditionscited by 42
  34. 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 42
  35. 2024Mora: Enabling Generalist Video Generation via A Multi-Agent Frameworkcited by 41
  36. 2023DragVideo: Interactive Drag-style Video Editingcited by 40
  37. 2024Video Editing via Factorized Diffusion Distillationcited by 40
  38. 2024Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slicescited by 40

50 most cited of 253 All 253 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editing
  2. 2023Video ControlNet: Towards Temporally Consistent Synthetic-to-Real Video Translation Using Conditional Image Diffusion Models
  3. 2023Structure and Content-Guided Video Synthesis with Diffusion Models
  4. 2024VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide
  5. 2023FlowZero: Zero-Shot Text-to-Video Synthesis with LLM-Driven Dynamic Scene Syntax
  6. 2023AnimateZero: Video Diffusion Models are Zero-Shot Image Animators
  7. 2023Style-A-Video: Agile Diffusion for Arbitrary Text-based Video Style Transfer
  8. 2023DiffusionAtlas: High-Fidelity Consistent Diffusion Video Editing