mlsanityData to 7 Oct 2026

Consistent Video-to-Video Transfer Using Synthetic Dataset

arXiv 2311.00213 ↗First posted 1 Nov 2023cs.CV cs.AIcited by 79

Not on the map. Why?

Cited by 27 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a novel and efficient approach for text-based video-to-video editing that eliminates the need for resource-intensive per-video-per-model finetuning. At the core of our approach is a synthetic paired video dataset tailored for video-to-video transfer tasks. Inspired by Instruct Pix2Pix's image transfer via editing instruction, we adapt this paradigm to the video domain. Extending the Prompt-to-Prompt to videos, we efficiently generate paired samples, each with an input video and its edited counterpart. Alongside this, we introduce the Long Video Sampling Correction during sampling, ensuring consistent long videos across batches. Our method surpasses current methods like Tune-A-Video, heralding substantial progress in text-based video-to-video editing and suggesting exciting avenues for further exploration and deployment.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Builds on 56 of its 61 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

56 of 61 references.

References · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
38 more
  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  2. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  9. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  10. 2022Video Diffusion Modelscited by 2,280
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  13. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  14. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  15. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  16. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  17. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  18. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  19. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  20. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  21. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  22. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  23. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  24. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  25. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  26. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  27. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  28. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  29. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  30. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  31. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  32. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  33. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  34. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  35. 2022Novel View Synthesis with Diffusion Modelscited by 299
  36. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  37. 2022Inversion-Based Style Transfer with Diffusion Modelscited by 290
  38. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 287

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 12; 2025: 30; 2026: 36.

79 papers.

Sort:

By area:

  1. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 151
  2. 2025Scaling Instruction-Based Video Editing with a High-Quality Synthetic Datasetcited by 76
  3. 2024ReVideo: Remake a Video with Motion and Content Controlcited by 74
  4. 2025InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Constructioncited by 62
  5. 2025Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialistscited by 61
  6. 2025EditVerse: Unifying Image and Video Editing and Generation with In-Context Learningcited by 53
  7. 2025OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editingcited by 50
  8. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  9. 2025MiniMax-Remover: Taming Bad Noise Helps Video Object Removalcited by 42
  10. 2024Video Editing via Factorized Diffusion Distillationcited by 40
  11. 2024Generative Video Propagationcited by 36
  12. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
38 more
  1. 2025In-Context Learning with Unpaired Clips for Instruction-based Video Editingcited by 34
  2. 2026Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidancecited by 34
  3. 2025InstructX: Towards Unified Visual Editing with MLLM Guidancecited by 31
  4. 2025RelightVid: Temporal-Consistent Diffusion Model for Video Relightingcited by 28
  5. 2025IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessmentcited by 27
  6. 2024VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editingcited by 24
  7. 2025Region-Constraint In-Context Generation for Instructional Video Editingcited by 22
  8. 2023VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Modelscited by 19
  9. 2025VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimizationcited by 16
  10. 2025SketchVideo: Sketch-based Video Generation and Editingcited by 15
  11. 2026LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editingcited by 15
  12. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  13. 2025JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generationcited by 12
  14. 2025EasyV2V: A High-quality Instruction-based Video Editing Frameworkcited by 11
  15. 2026Bernini: Latent Semantic Planning for Video Diffusioncited by 11
  16. 2025DreamVE: Unified Instruction-based Image and Video Editingcited by 9
  17. 2024Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generationcited by 8
  18. 2025RelightMaster: Precise Video Relighting with Multi-plane Light Imagescited by 8
  19. 2026Coarse-to-Real: Generative Rendering for Populated Dynamic Scenescited by 8
  20. 2026SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editingcited by 8
  21. 2026StreamingEffect: Real-Time Human-Centric Video Effect Generationcited by 8
  22. 2024NaRCan: Natural Refined Canonical Image with Integration of Diffusion Prior for Video Editingcited by 7
  23. 2025InstructVEdit: A Holistic Approach for Instructional Video Editingcited by 7
  24. 2025EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editingcited by 7
  25. 2025IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generationcited by 6
  26. 2025Towards Reason-Informed Video Editing in Unified Models with Self-Reflective Learningcited by 6
  27. 2026Evolution of Video Generative Foundationscited by 6
  28. 2025AdaFlow: Efficient Long Video Editing via Adaptive Attention Slimming And Keyframe Selectioncited by 5
  29. 2025VideoCoF: Unified Video Editing with Temporal Reasonercited by 5
  30. 2026FFP-300K: Scaling First-Frame Propagation for Generalizable Video Editingcited by 5
  31. 2026Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editingcited by 5
  32. 2025IC-Effect: Precise and Efficient Video Effects Editing via In-Context Learningcited by 4
  33. 2024Dense-Face: Personalized Face Generation Model via Dense Annotation Predictioncited by 3
  34. 2026Tuning-free Visual Effect Transfer across Videoscited by 3
  35. 2026LightCtrl: Training-free Controllable Video Relightingcited by 3
  36. 2026LIVEditor-14B: Lightning Unified Video Editing via In-Context Sparse Attentioncited by 3
  37. 2026SteerVTE: Seamless Video Text Editing with Style and Glyph Controlcited by 3
  38. 2026CoT-Edit: Let CoT Guide Instruction Video Editingcited by 3

50 most cited of 79 All 79 (JSON)