mlsanityData to 14 Oct 2026

InstructVid2Vid: Controllable Video Editing with Natural Language Instructions

arXiv 2305.12328 ↗First posted 21 May 2023cs.CV cs.AI cs.MMcited by 44

Not on the map. Why?

Abstract (arXiv)

We introduce InstructVid2Vid, an end-to-end diffusion-based methodology for video editing guided by human language instructions. Our approach empowers video manipulation guided by natural language directives, eliminating the need for per-example fine-tuning or inversion. The proposed InstructVid2Vid model modifies a pretrained image generation model, Stable Diffusion, to generate a time-dependent sequence of video frames. By harnessing the collective intelligence of disparate models, we engineer a training dataset rich in video-instruction triplets, which is a more cost-efficient alternative to collecting data in real-world scenarios. To enhance the coherence between successive frames within the generated videos, we propose the Inter-Frames Consistency Loss and incorporate it during the training process. With multimodal classifier-free guidance during the inference stage, the generated videos is able to resonate with both the input video and the accompanying instructions. Experimental results demonstrate that InstructVid2Vid is capable of generating high-quality, temporally coherent videos and performing diverse edits, including attribute editing, background changes, and style transfer. These results underscore the versatility and effectiveness of our proposed method.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 51 of its 62 references that are in the map. First posted on arXiv 21 May 2023.

Builds on

51 of 62 references.

References · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  8. 2017Neural Discrete Representation Learningcited by 4,712
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
38 more
  1. 2022Video Diffusion Modelscited by 2,305
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  5. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,476
  6. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  7. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
  8. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  9. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  10. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  11. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  12. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  13. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  14. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
  15. 2023Pix2Video: Video Editing using Image Diffusioncited by 320
  16. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 314
  17. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  18. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 285
  19. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  20. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  21. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 191
  22. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 154
  23. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  24. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Modelscited by 137
  25. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  26. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 116
  27. 2023Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Modelscited by 99
  28. 2023DiffusionRig: Learning Personalized Priors for Facial Appearance Editingcited by 75
  29. 2022Fine-Grained Semantically Aligned Vision-Language Pre-Trainingcited by 68
  30. 2023Edit-A-Video: Single Video Editing with Object-Aware Consistencycited by 67
  31. 2023Guided Image Synthesis via Initial Image Editing in Diffusion Modelcited by 63
  32. 2023Improving Diffusion Models for Scene Text Editing with Dual Encoderscited by 45
  33. 2023Make-A-Protagonist: Generic Video Editing with An Ensemble of Expertscited by 34
  34. 2023Edit Everything: A Text-Guided Generative System for Images Editingcited by 33
  35. 2023iEdit: Localised Text-guided Image Editing with Weak Supervisioncited by 20
  36. 2023DialogPaint: A Dialog-based Image Editing Modelcited by 8
  37. 2023Towards Real-time Text-driven Image Manipulation with Unconditional Diffusion Modelscited by 7
  38. 2022DBA: Efficient Transformer with Dynamic Bilinear Low-Rank Attentioncited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 8; 2025: 12; 2026: 19.

44 papers.

Sort:

By area:

  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  2. 2023A Survey on Video Diffusion Modelscited by 206
  3. 2025Scaling Instruction-Based Video Editing with a High-Quality Synthetic Datasetcited by 79
  4. 2025InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Constructioncited by 62
  5. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  6. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  7. 2025In-Context Learning with Unpaired Clips for Instruction-based Video Editingcited by 35
  8. 2024VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editingcited by 25
  9. 2023VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Modelscited by 19
  10. 2023Fine-grained Controllable Video Generation via Object Appearance and Contextcited by 18
  11. 2025FiVE: A Fine-grained Video Editing Benchmark for Evaluating Emerging Diffusion and Rectified Flow Modelscited by 16
  12. 2023Dancing Avatar: Pose and Text-Guided Human Motion Videos Synthesis with Image Diffusion Modelcited by 13
32 more
  1. 2025EasyV2V: A High-quality Instruction-based Video Editing Frameworkcited by 13
  2. 2024Portrait Video Editing Empowered by Multimodal Generative Priorscited by 12
  3. 2025TC-Light: Temporally Coherent Generative Rendering for Realistic World Transfercited by 11
  4. 2025Tinker: Diffusion's Gift to 3D--Multi-View Consistent Editing From Sparse Inputs without Per-Scene Optimizationcited by 9
  5. 2026VISTA: Triplet-Supervised Video Style Transfer with Diffusion Transformerscited by 8
  6. 2026SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editingcited by 8
  7. 2023Testing the Depth of ChatGPT's Comprehension via Cross-Modal Tasks Based on ASCII-Art: GPT3.5's Abilities in Regard to Recognizing and Generating ASCII-Art Are Not Totally Lackingcited by 7
  8. 2025Towards Reason-Informed Video Editing in Unified Models with Self-Reflective Learningcited by 7
  9. 2026Evolution of Video Generative Foundationscited by 7
  10. 2025Prompt-Driven Agentic Video Editing System: Autonomous Comprehension of Long-Form, Story-Driven Mediacited by 6
  11. 2024Generative Timelines for Instructed Visual Assemblycited by 5
  12. 2026InstructAV2AV: Instruction-Guided Audio-Video Joint Editingcited by 5
  13. 2025Counterfactual World Models via Digital Twin-conditioned Video Diffusioncited by 4
  14. 2026NOVA: Sparse Control, Dense Synthesis for Pair-Free Video Editingcited by 4
  15. 2025Text-Driven Reasoning Video Editing via Reinforcement Learning on Digital Twin Representationscited by 3
  16. 2026Memory-V2V: Memory-Augmented Video-to-Video Diffusion for Consistent Multi-Turn Editingcited by 3
  17. 2026ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationshipscited by 3
  18. 2026CoT-Edit: Let CoT Guide Instruction Video Editingcited by 3
  19. 2026Instruction-Based Video Editing by Repurposing an Image Editing Modelcited by 3
  20. 2024VIA: Unified Spatiotemporal Video Adaptation Framework for Global and Local Video Editingcited by 2
  21. 2024Beyond Generation: Unlocking Universal Editing via Self-Supervised Fine-Tuningcited by 2
  22. 2025SneakPeek: Future-Guided Instructional Streaming Video Generationcited by 1
  23. 2026PLACID: Identity-Preserving Multi-Object Compositing via Video Diffusion with Synthetic Trajectoriescited by 1
  24. 2026RealMaster: Lifting Rendered Scenes into Photorealistic Videocited by 1
  25. 2026InsEdit: Towards Instruction-based Visual Editing via Data-Efficient Video Diffusion Models Adaptationcited by 1
  26. 2026EgoPlay: Event-Triggered Video Editing for Egocentric Streamscited by 1
  27. 2026BeyondMasks: Evaluating Causal and Physical Consistency in Video Object Removalcited by 1
  28. 2026UniEditBench: A Unified and Cost-Effective Benchmark for Image and Video Editing via Distilled MLLMscited by 0
  29. 2026Sound Sparks Motion: Audio and Text Tuning for Video Editingcited by 0
  30. 2026Reasoning to Align: Implicit Reasoning in Diffusion Transformers for Video Editingcited by 0
  31. 2026From Dense Prediction to Visual Editing: Structured Supervision for Unified Image and Video Creationcited by 0
  32. 2026Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Are Image-to-Video Models Good Zero-Shot Image Editors?
  2. 2025InstructX: Towards Unified Visual Editing with MLLM Guidance
  3. 2026Omni-Video 2: Scaling MLLM-Conditioned Diffusion for Unified Video Generation and Editing
  4. 2025VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization
  5. 2026Tele-Omni: a Unified Multimodal Framework for Video Generation and Editing
  6. 2026ThinkV2V: Unleashing the Reasoning Capability of MLLMs for Instruction-Guided Video Editing
  7. 2024Instruction-based Image Manipulation by Watching How Things Move
  8. 2025Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learning