# Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance

arXiv: https://arxiv.org/abs/2603.02175 · HTML: https://mlsanity.com/paper/2603.02175/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 36 references that are in the map.
- First posted on arXiv 2 Mar 2026.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [Transfer between Modalities with MetaQueries](https://mlsanity.com/paper/2504.06256/) (2025)
- [FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing](https://mlsanity.com/paper/2310.05922/) (2023)
- [AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks](https://mlsanity.com/paper/2403.14468/) (2024)
- [RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models](https://mlsanity.com/paper/2312.04524/) (2023)
- [UniVideo: Unified Understanding, Generation, and Editing for Videos](https://mlsanity.com/paper/2510.08377/) (2025)
- [Consistent Video-to-Video Transfer Using Synthetic Dataset](https://mlsanity.com/paper/2311.00213/) (2023)
- [Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset](https://mlsanity.com/paper/2510.15742/) (2025)
- [Kling-Omni Technical Report](https://mlsanity.com/paper/2512.16776/) (2025)
- [GPT-IMAGE-EDIT-1.5M: A Million-Scale, GPT-Generated Image Dataset](https://mlsanity.com/paper/2507.21033/) (2025)
- [EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modeling](https://mlsanity.com/paper/2509.23909/) (2025)
- [InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Construction](https://mlsanity.com/paper/2503.20287/) (2025)
- [Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists](https://mlsanity.com/paper/2502.06734/) (2025)
- [OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing](https://mlsanity.com/paper/2512.07826/) (2025)
- [Omni-Video: Democratizing Unified Video Understanding and Generation](https://mlsanity.com/paper/2507.06119/) (2025)

## Built on by (most cited first, 30 of 36)

- [Aurora: Unified Video Editing with a Tool-Using Agent](https://mlsanity.com/paper/2605.18748/) (2026)
- [StreamingEffect: Real-Time Human-Centric Video Effect Generation](https://mlsanity.com/paper/2605.17019/) (2026)
- [LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing](https://mlsanity.com/paper/2606.06042/) (2026)
- [JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion](https://mlsanity.com/paper/2608.03974/) (2026)
- [VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing](https://mlsanity.com/paper/2608.05485/) (2026)
- [Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation](https://mlsanity.com/paper/2609.11638/) (2026)
- [Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoE](https://mlsanity.com/paper/2605.02641/) (2026)
- [Smart-Insertion-V: Photorealistic Video Insertion via a Closed-Loop Feedback Dual-Stream Framework](https://mlsanity.com/paper/2605.23891/) (2026)
- [CoVEBench: Can Video Editing Models Handle Complex Instructions?](https://mlsanity.com/paper/2606.08415/) (2026)
- [SteerVTE: Seamless Video Text Editing with Style and Glyph Control](https://mlsanity.com/paper/2606.23254/) (2026)
- [ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationships](https://mlsanity.com/paper/2607.14681/) (2026)
- [JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation](https://mlsanity.com/paper/2606.03168/) (2026)
- [TIDE: Task-Isolated Diffusion for Unified Video Editing and Generation](https://mlsanity.com/paper/2606.08260/) (2026)
- [GRNEdit: Efficient General Video Editing from a New Binary-Evidence Perspective in Generative Refinement Networks](https://mlsanity.com/paper/2608.16328/) (2026)
- [CoinVE-200K: A Large-Scale High-Quality Dataset for Compositional Instruction-Guided Video Editing](https://mlsanity.com/paper/2608.17566/) (2026)
- [EditStream: A Unified Autoregressive Framework for Interactive Video Generation and Editing](https://mlsanity.com/paper/2608.21424/) (2026)
- [Sparkle: Realizing Lively Instruction-Guided Video Background Replacement via Decoupled Guidance](https://mlsanity.com/paper/2605.06535/) (2026)
- [Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training](https://mlsanity.com/paper/2608.24680/) (2026)
- [ImVideoEdit: Image-learning Video Editing via 2D Spatial Difference Attention Blocks](https://mlsanity.com/paper/2604.07958/) (2026)
- [What Semantics Survive the Connector? Diagnosing VLM-to-DiT Alignment in Video Editing](https://mlsanity.com/paper/2605.20795/) (2026)
- [Reasoning to Align: Implicit Reasoning in Diffusion Transformers for Video Editing](https://mlsanity.com/paper/2605.24674/) (2026)
- [RefCaptioner: Multi-Reference Image-Grounded Video Captioning](https://mlsanity.com/paper/2607.28509/) (2026)
- [MoRoute: Dynamic Routing for In-Context Multimodal Video Generation](https://mlsanity.com/paper/2607.29545/) (2026)
- [ContextMaster: Interactive Multi-Shot Video Creation via Fixed-Budget Sparse Context Routing](https://mlsanity.com/paper/2608.04956/) (2026)
- [EffectLearner: World-Aware Object-Effect Reasoning for Real-World Video Object Removal](https://mlsanity.com/paper/2608.05565/) (2026)
- [VicEdit: Learning to Edit Videos from Visual In-Context Examples](https://mlsanity.com/paper/2608.16745/) (2026)
- [InfinityEdit: Infinite Video Editing with a Lightweight Edit-Ignition Adapter](https://mlsanity.com/paper/2608.20910/) (2026)
- [RefVideo-6M: A Reliable Reference-Based Dataset for Instructional Video Editing](https://mlsanity.com/paper/2608.26101/) (2026)
- [One Editor, Many Edits: A Unified Training-Free Framework for Diverse Video Editing](https://mlsanity.com/paper/2609.04190/) (2026)
- [Enhanced Video Text Editing with Trajectory-Aligned Glyph Rendering](https://mlsanity.com/paper/2609.34178/) (2026)

## Abstract (arXiv)

Instruction-based video editing has witnessed rapid progress, yet current methods often struggle with precise visual control, as natural language is inherently limited in describing complex visual nuances. Although reference-guided editing offers a robust solution, its potential is currently bottlenecked by the scarcity of high-quality paired training data. To bridge this gap, we introduce a scalable data generation pipeline that transforms existing video editing pairs into high-fidelity training quadruplets, leveraging image generative models to create synthesized reference scaffolds. Using this pipeline, we construct RefVIE, a large-scale dataset tailored for instruction-reference-following tasks, and establish RefVIE-Bench for comprehensive evaluation. Furthermore, we propose a unified editing architecture, Kiwi-Edit, that synergizes learnable queries and latent visual features for reference semantic guidance. Our model achieves significant gains in instruction following and reference fidelity via a progressive multi-stage training curriculum. Extensive experiments demonstrate that our data and architecture establish a new state-of-the-art in controllable video editing. All datasets, models, and code is released at https://github.com/showlab/Kiwi-Edit.