Instruction-Based Video Editing by Repurposing an Image Editing Model
On the map: Diffusion models → Video generation
Abstract (arXiv)
Instruction-based video editing is commonly built on video-pretrained generative backbones: a video diffusion transformer is adapted, at considerable cost, to condition on a source video and an editing instruction. In this report we explore a different route and show that a strong instruction-based image editing model can edit videos by operating directly on video-VAE latents. Starting from Qwen-Image-Edit, we arrange the latent frames of a Wan~2.1 video VAE as tiles of one large virtual image, reuse the editor's image positional encoding for every tile, and bridge the two latent spaces with a pair of lightweight input/output projections warm-started from the editor's own patchify and unpatchify layers, so that at initialization a (static) video is embedded exactly as an image the model already understands. The whole system is then fine-tuned on the public Ditto-1M editing triplets, and a few denoising steps of Wan~2.2 serve as an optional temporal enhancer. We motivate the design with a chain of zero-training observations: the stock image editor already edits a video presented as a contact sheet; it is indifferent to whether the sheet's tokens come from one joint encode or from per-frame encodes stitched in latent space; and it even edits genuine video latents zero-shot to a clearly recognizable degree, leaving fine-tuning only a fidelity gap to close. Our results suggest that, despite the large investment in training video latent spaces, per-frame video latents remain close enough to the image domain that mature image editing priors transfer with minimal adaptation. Project Page: https://yunpeng1998.github.io/Qwen-Video-Edit-Page Code: https://github.com/yunpeng1998/Qwen-Video-Edit Model: https://huggingface.co/yunpeng1998/Qwen-Video-Edit
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 37 of its 40 references that are in the map. First posted on arXiv 14 Aug 2026.
Builds on
37 of 40 references.
Closest ancestors · 11
Same direction, newest first
- 2026SANA-Streaming: Real-time Streaming Video Editing with Hybrid Diffusion Transformer28 May
- 2025StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation10 Nov
- 2025VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation18 Mar
- 2025VACE: All-in-One Video Creation and Editing10 Mar
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
- 2024EffiVED:Efficient Video Editing via Text-instruction Diffusion Models18 Mar
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
- 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editing18 Aug
- 2023Pix2Video: Video Editing using Image Diffusion22 Mar
- 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
- 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
Other references · 26
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2022Scalable Diffusion Models with Transformerscited by 6,054
- 2022Flow Matching for Generative Modelingcited by 5,702
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
- 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
- 2025Qwen-Image Technical Reportcited by 1,044
- 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
- 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 442
- 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 429
- 2023Video-P2P: Video Editing with Cross-attention Controlcited by 314
14 more
- 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 154
- 2023CoDeF: Content Deformation Fields for Temporally Consistent Video Processingcited by 108
- 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 87
- 2023Consistent Video-to-Video Transfer Using Synthetic Datasetcited by 81
- 2025Scaling Instruction-Based Video Editing with a High-Quality Synthetic Datasetcited by 79
- 2025InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Constructioncited by 62
- 2025Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialistscited by 61
- 2025EditVerse: Unifying Image and Video Editing and Generation with In-Context Learningcited by 54
- 2023InstructVid2Vid: Controllable Video Editing with Natural Language Instructionscited by 44
- 2026LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editingcited by 15
- 2025DreamVE: Unified Instruction-based Image and Video Editingcited by 11
- 2025InstructVEdit: A Holistic Approach for Instructional Video Editingcited by 7
- 2026HP-Edit: A Human-Preference Post-Training Framework for Image Editingcited by 6
- 2026JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusioncited by 6
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026DuoMatching: Joint-Marginal Distribution Matching for Few-Step Video Generationcited by 1
- 2026One Editor, Many Edits: A Unified Training-Free Framework for Diverse Video Editingcited by 0
- 2026OneFixer: High-Quality and Consistent One-Step Autoregressive 3DGS Refinement for Driving Scenescited by 0
- 2026DuoMatching: Joint-Marginal Distribution Matching for Few-Step Video Generation2 Oct 2026
- 2026OneFixer: High-Quality and Consistent One-Step Autoregressive 3DGS Refinement for Driving Scenes26 Sep 2026
- 2026One Editor, Many Edits: A Unified Training-Free Framework for Diverse Video Editing3 Sep 2026
Similar, not cited
Close in wording, no citation either way
- 2026Transforming Image Editors into Video Editors
- 2025VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization
- 2026InsEdit: Towards Instruction-based Visual Editing via Data-Efficient Video Diffusion Models Adaptation
- 2026VIBE: Visual Instruction Based Editor
- 2025V2Edit: Versatile Video Diffusion Editor for Videos and 3D Scenes
- 2025Are Image-to-Video Models Good Zero-Shot Image Editors?
- 2026Edit2Interp: Adapting Image Foundation Models from Spatial Editing to Video Frame Interpolation with Few-Shot Learning
- 2024Video Editing via Factorized Diffusion Distillation