# Aurora: Unified Video Editing with a Tool-Using Agent

arXiv: https://arxiv.org/abs/2605.18748 · HTML: https://mlsanity.com/paper/2605.18748/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 37 of its 43 references that are in the map.
- First posted on arXiv 18 May 2026.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 37)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer](https://mlsanity.com/paper/2511.22699/) (2025)
- [UltraEdit: Instruction-based Fine-Grained Image Editing at Scale](https://mlsanity.com/paper/2407.05282/) (2024)
- [OmniGen2: Towards Instruction-Aligned Multimodal Generation](https://mlsanity.com/paper/2506.18871/) (2025)
- [Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer](https://mlsanity.com/paper/2311.17009/) (2023)
- [UniVideo: Unified Understanding, Generation, and Editing for Videos](https://mlsanity.com/paper/2510.08377/) (2025)
- [Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset](https://mlsanity.com/paper/2510.15742/) (2025)
- [SpatialVID: A Large-Scale Video Dataset with Spatial Annotations](https://mlsanity.com/paper/2509.09676/) (2025)
- [OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation](https://mlsanity.com/paper/2505.20292/) (2025)
- [InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Construction](https://mlsanity.com/paper/2503.20287/) (2025)
- [Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists](https://mlsanity.com/paper/2502.06734/) (2025)
- [EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning](https://mlsanity.com/paper/2509.20360/) (2025)
- [OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing](https://mlsanity.com/paper/2512.07826/) (2025)
- [HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioning](https://mlsanity.com/paper/2509.08519/) (2025)
- [ROSE: Remove Objects with Side Effects in Videos](https://mlsanity.com/paper/2508.18633/) (2025)
- [Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance](https://mlsanity.com/paper/2603.02175/) (2026)
- [JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization](https://mlsanity.com/paper/2511.23002/) (2025)
- [Gen-Searcher: Reinforcing Agentic Search for Image Generation](https://mlsanity.com/paper/2603.28767/) (2026)
- [InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing](https://mlsanity.com/paper/2603.09877/) (2026)
- [Region-Constraint In-Context Generation for Instructional Video Editing](https://mlsanity.com/paper/2512.17650/) (2025)
- [Agent Banana: High-Fidelity Image Editing with Agentic Thinking and Tooling](https://mlsanity.com/paper/2602.09084/) (2026)
- [GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaboration](https://mlsanity.com/paper/2412.04440/) (2024)
- [MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Models](https://mlsanity.com/paper/2505.19415/) (2025)

## Built on by (most cited first, 9 of 9)

- [VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing](https://mlsanity.com/paper/2608.05485/) (2026)
- [Agent Skills Should Go Beyond Text: The Case for Visual Skills](https://mlsanity.com/paper/2606.01414/) (2026)
- [Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training](https://mlsanity.com/paper/2608.24680/) (2026)
- [Agentic Visual Generation: From Generative Models to Agentic Control](https://mlsanity.com/paper/2609.06758/) (2026)
- [EpiCon: Collective Agent Learning through Co-Evolving Multimodal Memory](https://mlsanity.com/paper/2609.37923/) (2026)
- [ContextMaster: Interactive Multi-Shot Video Creation via Fixed-Budget Sparse Context Routing](https://mlsanity.com/paper/2608.04956/) (2026)
- [Thinking on Shots: Consistent Multi-Shot Video Editing with Agentic Reasoning](https://mlsanity.com/paper/2608.26809/) (2026)
- [ALIVE: Interaction-Aligned Object Insertion for First-Frame-Guided Video Editing](https://mlsanity.com/paper/2610.08779/) (2026)
- [VIEWCRAFT: Evolving What Multimodal Agents See and How They Read It](https://mlsanity.com/paper/2610.15915/) (2026)

## Abstract (arXiv)

Recent video editing models have converged on a unified conditioning design: a single diffusion transformer jointly consumes text, source video, and reference images, and one set of weights covers replacement, removal, style transfer, and reference-driven insertion. The design is flexible, but it assumes that the user already provides model-ready text, reference images, and spatial grounding for local edits, which real requests often omit. We present Aurora, an agentic video editing framework that pairs a tool-augmented vision-language model (VLM) agent with a unified video diffusion transformer. The VLM agent maps a raw user request to a structured edit plan aligned with the transformer's conditioning channels, thereby resolving textual and visual underspecification before generation. We train the VLM agent with supervised data for complete edit planning and reference-image selection, together with preference pairs for robust tool use and instruction refinement. We introduce AgentEdit-Bench to evaluate agent-enhanced video editing under textual and visual underspecification. Experiments on AgentEdit-Bench and two existing video editing benchmarks show that Aurora improves over instruction-only baselines and that the VLM agent transfers to compatible frozen video editing models. Project page: https://yeates.github.io/Aurora-Page