# EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning

arXiv: https://arxiv.org/abs/2509.20360 · HTML: https://mlsanity.com/paper/2509.20360/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 72 of its 83 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Builds on (most cited first, 30 of 72)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model](https://mlsanity.com/paper/2408.11039/) (2024)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)

## Built on by (most cited first, 30 of 54)

- [UniVideo: Unified Understanding, Generation, and Editing for Videos](https://mlsanity.com/paper/2510.08377/) (2025)
- [Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset](https://mlsanity.com/paper/2510.15742/) (2025)
- [OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing](https://mlsanity.com/paper/2512.07826/) (2025)
- [Rethinking Video Generation Model for the Embodied World](https://mlsanity.com/paper/2601.15282/) (2026)
- [PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generation](https://mlsanity.com/paper/2512.24551/) (2025)
- [Omni-Video 2: Scaling MLLM-Conditioned Diffusion for Unified Video Generation and Editing](https://mlsanity.com/paper/2602.08820/) (2026)
- [VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization](https://mlsanity.com/paper/2512.16906/) (2025)
- [LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing](https://mlsanity.com/paper/2606.26740/) (2026)
- [EasyV2V: A High-quality Instruction-based Video Editing Framework](https://mlsanity.com/paper/2512.16920/) (2025)
- [Lance: Unified Multimodal Modeling by Multi-Task Synergy](https://mlsanity.com/paper/2605.18678/) (2026)
- [Bernini: Latent Semantic Planning for Video Diffusion](https://mlsanity.com/paper/2605.22344/) (2026)
- [Aurora: Unified Video Editing with a Tool-Using Agent](https://mlsanity.com/paper/2605.18748/) (2026)
- [EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing](https://mlsanity.com/paper/2512.06065/) (2025)
- [VideoCoF: Unified Video Editing with Temporal Reasoner](https://mlsanity.com/paper/2512.07469/) (2025)
- [SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing](https://mlsanity.com/paper/2603.19228/) (2026)
- [SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editing](https://mlsanity.com/paper/2605.25193/) (2026)
- [RoboVIP: Multi-View Video Generation with Visual Identity Prompting Augments Robot Manipulation](https://mlsanity.com/paper/2601.05241/) (2026)
- [Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing](https://mlsanity.com/paper/2606.30599/) (2026)
- [LoVoRA: Text-guided and Mask-free Video Object Removal and Addition with Learnable Object-aware Localization](https://mlsanity.com/paper/2512.02933/) (2025)
- [FFP-300K: Scaling First-Frame Propagation for Generalizable Video Editing](https://mlsanity.com/paper/2601.01720/) (2026)
- [VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing](https://mlsanity.com/paper/2608.05485/) (2026)
- [LIVEditor-14B: Lightning Unified Video Editing via In-Context Sparse Attention](https://mlsanity.com/paper/2605.04569/) (2026)
- [Memory-V2V: Memory-Augmented Video-to-Video Diffusion for Consistent Multi-Turn Editing](https://mlsanity.com/paper/2601.16296/) (2026)
- [Activation Steering of Video Generation Models via Reduced-Order Linear Optimal Control](https://mlsanity.com/paper/2606.04775/) (2026)
- [Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers](https://mlsanity.com/paper/2607.28611/) (2026)
- [Instruction-Based Video Editing by Repurposing an Image Editing Model](https://mlsanity.com/paper/2608.14790/) (2026)
- [ViDiC: Video Difference Captioning](https://mlsanity.com/paper/2512.03405/) (2025)
- [CreativeVR: Diffusion-Prior-Guided Approach for Structure and Motion Restoration in Generative and Real Videos](https://mlsanity.com/paper/2512.12060/) (2025)
- [ViFeEdit: A Video-Free Tuner of Your Video Diffusion Transformer](https://mlsanity.com/paper/2603.15478/) (2026)
- [EditStream: A Unified Autoregressive Framework for Interactive Video Generation and Editing](https://mlsanity.com/paper/2608.21424/) (2026)

## Abstract (arXiv)

Recent advances in foundation models highlight a clear trend toward unification and scaling, showing emergent capabilities across diverse domains. While image generation and editing have rapidly transitioned from task-specific to unified frameworks, video generation and editing remain fragmented due to architectural limitations and data scarcity. In this work, we introduce EditVerse, a unified framework for image and video generation and editing within a single model. By representing all modalities, i.e., text, image, and video, as a unified token sequence, EditVerse leverages self-attention to achieve robust in-context learning, natural cross-modal knowledge transfer, and flexible handling of inputs and outputs with arbitrary resolutions and durations. To address the lack of video editing training data, we design a scalable data pipeline that curates 232K video editing samples and combines them with large-scale image and video datasets for joint training. Furthermore, we present EditVerseBench, the first benchmark for instruction-based video editing covering diverse tasks and resolutions. Extensive experiments and user studies demonstrate that EditVerse achieves state-of-the-art performance, surpassing existing open-source and commercial models, while exhibiting emergent editing and generation abilities across modalities.