# UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalist arXiv: https://arxiv.org/abs/2511.08521 ยท HTML: https://mlsanity.com/paper/2511.08521/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. - Builds on 53 of its 60 references that are in the map. - First posted on arXiv 11 Nov 2025. ## Placed in - LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/) ## Builds on (most cited first, 30 of 53) - [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024) - [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025) - [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022) - [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022) - [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024) - [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023) - [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023) - [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023) - [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022) - [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023) - [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023) - [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021) - [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023) - [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024) - [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023) - [ShareGPT4Video: Improving Video Understanding and Generation with Better Captions](https://mlsanity.com/paper/2406.04325/) (2024) - [Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding](https://mlsanity.com/paper/2311.08046/) (2023) - [VideoAgent: Long-form Video Understanding with Large Language Model as Agent](https://mlsanity.com/paper/2403.10517/) (2024) - [Video-P2P: Video Editing with Cross-attention Control](https://mlsanity.com/paper/2303.04761/) (2023) - [Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos](https://mlsanity.com/paper/2304.01186/) (2023) - [VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation](https://mlsanity.com/paper/2409.04429/) (2024) - [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023) - [Seedance 1.0: Exploring the Boundaries of Video Generation Models](https://mlsanity.com/paper/2506.09113/) (2025) - [Show-o2: Improved Native Unified Multimodal Models](https://mlsanity.com/paper/2506.15564/) (2025) - [VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding](https://mlsanity.com/paper/2403.11481/) (2024) - [Self-Chained Image-Language Model for Video Localization and Question Answering](https://mlsanity.com/paper/2305.06988/) (2023) - [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023) - [Conditional Image-to-Video Generation with Latent Flow Diffusion Models](https://mlsanity.com/paper/2303.13744/) (2023) - [Hierarchical Conditional Relation Networks for Video Question Answering](https://mlsanity.com/paper/2002.10698/) (2020) - [Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos](https://mlsanity.com/paper/2501.04001/) (2025) ## Built on by (most cited first, 15 of 15) - [Aurora: Unified Video Editing with a Tool-Using Agent](https://mlsanity.com/paper/2605.18748/) (2026) - [ShotVerse: Advancing Cinematic Camera Control for Text-Driven Multi-Shot Video Creation](https://mlsanity.com/paper/2603.11421/) (2026) - [AgenticVBench: Can AI Agents Complete Real-World Post-Production Tasks?](https://mlsanity.com/paper/2605.27705/) (2026) - [UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating](https://mlsanity.com/paper/2606.21661/) (2026) - [Better Call CineCrew: Consistent Ultra-Long Narrative-to-Film Generation](https://mlsanity.com/paper/2609.07720/) (2026) - [DramaDirector: Geometry-Guided Short Drama Generation](https://mlsanity.com/paper/2606.24107/) (2026) - [Plans You Can Check: Verifier-Grounded Learning of an Open-Weight Planner for Executable Video-Editing](https://mlsanity.com/paper/2608.25622/) (2026) - [Agentic Visual Generation: From Generative Models to Agentic Control](https://mlsanity.com/paper/2609.06758/) (2026) - [TOBench: A Task-Oriented Omni-Modal Benchmark for Real-World Tool-Using Agents](https://mlsanity.com/paper/2605.16909/) (2026) - [Thinking on Shots: Consistent Multi-Shot Video Editing with Agentic Reasoning](https://mlsanity.com/paper/2608.26809/) (2026) - [FRAMEWORKERS: A Dynamic Multi-Agent Framework for AI-Generated Video Production](https://mlsanity.com/paper/2608.29814/) (2026) - [DramaChain Bench: An End-to-End Benchmark for Short-Drama Generation](https://mlsanity.com/paper/2609.00646/) (2026) - [MVAgent: Multi-Agent Video Generation via Consistent Condition Construction and Shot-Level Policy Optimization](https://mlsanity.com/paper/2609.30609/) (2026) - [TemplateCraft: Agentic Visual Template Generation](https://mlsanity.com/paper/2609.31451/) (2026) - [VAMR: Multi-Question Agentic Reasoning for Efficient Long-Form Video Understanding](https://mlsanity.com/paper/2610.11171/) (2026) ## Abstract (arXiv) While specialized AI models excel at isolated video tasks like generation or understanding, real-world applications demand complex, iterative workflows that combine these capabilities. To bridge this gap, we introduce UniVA, an open-source, omni-capable multi-agent framework for next-generation video generalists that unifies video understanding, segmentation, editing, and generation into cohesive workflows. UniVA employs a Plan-and-Act dual-agent architecture that drives a highly automated and proactive workflow: a planner agent interprets user intentions and decomposes them into structured video-processing steps, while executor agents execute these through modular, MCP-based tool servers (for analysis, generation, editing, tracking, etc.). Through a hierarchical multi-level memory (global knowledge, task context, and user-specific preferences), UniVA sustains long-horizon reasoning, contextual continuity, and inter-agent communication, enabling interactive and self-reflective video creation with full traceability. This design enables iterative and any-conditioned video workflows (e.g., text/image/video-conditioned generation rightarrow multi-round editing rightarrow object segmentation rightarrow compositional synthesis) that were previously cumbersome to achieve with single-purpose models or monolithic video-language models. We also introduce UniVA-Bench, a benchmark suite of multi-step video tasks spanning understanding, editing, segmentation, and generation, to rigorously evaluate such agentic video systems. Both UniVA and UniVA-Bench are fully open-sourced, aiming to catalyze research on interactive, agentic, and general-purpose video intelligence for the next generation of multimodal AI systems. (https://univa.online/)