# Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Models

arXiv: https://arxiv.org/abs/2405.04233 · HTML: https://mlsanity.com/paper/2405.04233/

## Key facts

- Cited by 160 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 13 of its 16 references that are in the map.
- First posted on arXiv 7 May 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [All are Worth Words: A ViT Backbone for Diffusion Models](https://mlsanity.com/paper/2209.12152/) (2022)
- [Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning](https://mlsanity.com/paper/2311.10709/) (2023)
- [One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale](https://mlsanity.com/paper/2303.06555/) (2023)

## Built on by (most cited first, 30 of 160)

- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data](https://mlsanity.com/paper/2406.03736/) (2024)
- [Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis](https://mlsanity.com/paper/2412.04431/) (2024)
- [Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning](https://mlsanity.com/paper/2601.16163/) (2026)
- [Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling](https://mlsanity.com/paper/2409.02908/) (2024)
- [MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance](https://mlsanity.com/paper/2406.19680/) (2024)
- [Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation](https://mlsanity.com/paper/2602.02214/) (2026)
- [Tora: Trajectory-oriented Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2407.21705/) (2024)
- [Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrieval](https://mlsanity.com/paper/2506.03141/) (2025)
- [Identity-Preserving Text-to-Video Generation by Frequency Decomposition](https://mlsanity.com/paper/2411.17440/) (2024)
- [TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Models](https://mlsanity.com/paper/2503.05638/) (2025)
- [Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer](https://mlsanity.com/paper/2412.00733/) (2024)
- [SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference](https://mlsanity.com/paper/2502.18137/) (2025)
- [Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing](https://mlsanity.com/paper/2508.09192/) (2025)
- [IRASim: A Fine-Grained World Model for Robot Manipulation](https://mlsanity.com/paper/2406.14540/) (2024)
- [FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention](https://mlsanity.com/paper/2407.19918/) (2024)
- [StableAnimator: High-Quality Identity-Preserving Human Image Animation](https://mlsanity.com/paper/2411.17697/) (2024)
- [Pandora: Towards General World Model with Natural Language Actions and Video States](https://mlsanity.com/paper/2406.09455/) (2024)
- [Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency](https://mlsanity.com/paper/2510.08431/) (2025)
- [OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation](https://mlsanity.com/paper/2412.00115/) (2024)
- [OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation](https://mlsanity.com/paper/2505.20292/) (2025)
- [RewardDance: Reward Scaling in Visual Generation](https://mlsanity.com/paper/2509.08826/) (2025)
- [Goku: Flow Based Video Generative Foundation Models](https://mlsanity.com/paper/2502.04896/) (2025)
- [T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models](https://mlsanity.com/paper/2407.05965/) (2024)
- [RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers](https://mlsanity.com/paper/2502.15894/) (2025)
- [Vidar: Embodied Video Diffusion Model for Generalist Manipulation](https://mlsanity.com/paper/2507.12898/) (2025)
- [EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation](https://mlsanity.com/paper/2501.01895/) (2025)
- [WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model](https://mlsanity.com/paper/2411.17459/) (2024)
- [RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Control](https://mlsanity.com/paper/2502.10059/) (2025)

## Abstract (arXiv)

We introduce Vidu, a high-performance text-to-video generator that is capable of producing 1080p videos up to 16 seconds in a single generation. Vidu is a diffusion model with U-ViT as its backbone, which unlocks the scalability and the capability for handling long videos. Vidu exhibits strong coherence and dynamism, and is capable of generating both realistic and imaginative videos, as well as understanding some professional photography techniques, on par with Sora -- the most powerful reported text-to-video generator. Finally, we perform initial experiments on other controllable video generation, including canny-to-video generation, video prediction and subject-driven generation, which demonstrate promising results.