# UniVid: Unifying Vision Tasks with Pre-trained Video Generation Models

arXiv: https://arxiv.org/abs/2509.21760 · HTML: https://mlsanity.com/paper/2509.21760/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 39 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Builds on (most cited first, 30 of 35)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [VideoPoet: A Large Language Model for Zero-Shot Video Generation](https://mlsanity.com/paper/2312.14125/) (2023)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [OminiControl: Minimal and Universal Control for Diffusion Transformer](https://mlsanity.com/paper/2411.15098/) (2024)
- [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](https://mlsanity.com/paper/2212.02499/) (2022)
- [Visual Prompting via Image Inpainting](https://mlsanity.com/paper/2209.00647/) (2022)
- [Sequential Modeling Enables Scalable Learning for Large Vision Models](https://mlsanity.com/paper/2312.00785/) (2023)
- [TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation](https://mlsanity.com/paper/2412.03069/) (2024)
- [In-Context LoRA for Diffusion Transformers](https://mlsanity.com/paper/2410.23775/) (2024)
- [Long-Context Autoregressive Video Modeling with Next-Frame Prediction](https://mlsanity.com/paper/2503.19325/) (2025)
- [EasyControl: Adding Efficient and Flexible Control for Diffusion Transformer](https://mlsanity.com/paper/2503.07027/) (2025)
- [Dense Extreme Inception Network: Towards a Robust CNN Model for Edge Detection](https://mlsanity.com/paper/1909.01955/) (2019)
- [VDT: General-purpose Video Diffusion Transformers via Mask Modeling](https://mlsanity.com/paper/2305.13311/) (2023)
- [ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling](https://mlsanity.com/paper/2501.02487/) (2025)
- [Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists](https://mlsanity.com/paper/2502.06734/) (2025)

## Built on by (most cited first, 5 of 5)

- [VideoCoF: Unified Video Editing with Temporal Reasoner](https://mlsanity.com/paper/2512.07469/) (2025)
- [iMontage: Unified, Versatile, Highly Dynamic Many-to-many Image Generation](https://mlsanity.com/paper/2511.20635/) (2025)
- [IC-Effect: Precise and Efficient Video Effects Editing via In-Context Learning](https://mlsanity.com/paper/2512.15635/) (2025)
- [NOVA: Sparse Control, Dense Synthesis for Pair-Free Video Editing](https://mlsanity.com/paper/2603.02802/) (2026)
- [Lumos-Nexus: Efficient Frequency Bridging with Homogeneous Latent Space for Video Unified Models](https://mlsanity.com/paper/2605.31603/) (2026)

## Abstract (arXiv)

Large language models, trained on extensive corpora, successfully unify diverse linguistic tasks within a single generative framework. Inspired by this, recent works like Large Vision Model (LVM) extend this paradigm to vision by organizing tasks into sequential visual sentences, where visual prompts serve as the context to guide outputs. However, such modeling requires task-specific pre-training across modalities and sources, which is costly and limits scalability to unseen tasks. Given that pre-trained video generation models inherently capture temporal sequence dependencies, we explore a more unified and scalable alternative: can a pre-trained video generation model adapt to diverse image and video tasks? To answer this, we propose UniVid, a framework that fine-tunes a video diffusion transformer to handle various vision tasks without task-specific modifications. Tasks are represented as visual sentences, where the context sequence defines both the task and the expected output modality. We evaluate the generalization of UniVid from two perspectives: (1) cross-modal inference with contexts composed of both images and videos, extending beyond LVM's uni-modal setting; (2) cross-source tasks from natural to annotated data, without multi-source pre-training. Despite being trained solely on natural video data, UniVid generalizes well in both settings. Notably, understanding and generation tasks can easily switch by simply reversing the visual sentence order in this paradigm. These findings highlight the potential of pre-trained video generation models to serve as a scalable and unified foundation for vision modeling. Our code will be released at https://github.com/CUC-MIPG/UniVid.