# Video Diffusion Models: A Survey

arXiv: https://arxiv.org/abs/2405.03150 · HTML: https://mlsanity.com/paper/2405.03150/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 147 of its 178 references that are in the map.
- First posted on arXiv 6 May 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 147)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)

## Built on by (most cited first, 30 of 30)

- [Fr'echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos](https://mlsanity.com/paper/2407.16124/) (2024)
- [Efficient Diffusion Models: A Survey](https://mlsanity.com/paper/2502.06805/) (2025)
- [MoFlow: One-Step Flow Matching for Human Trajectory Forecasting via Implicit Maximum Likelihood Estimation based Distillation](https://mlsanity.com/paper/2503.09950/) (2025)
- [Exploring the Evolution of Physics Cognition in Video Generation: A Survey](https://mlsanity.com/paper/2503.21765/) (2025)
- [UniVid: The Open-Source Unified Video Model](https://mlsanity.com/paper/2509.24200/) (2025)
- [Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer](https://mlsanity.com/paper/2509.24899/) (2025)
- [Simulating the Visual World with Artificial Intelligence: A Roadmap](https://mlsanity.com/paper/2511.08585/) (2025)
- [Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directions](https://mlsanity.com/paper/2601.07823/) (2026)
- [Neodragon: Mobile Video Generation using Diffusion Transformer](https://mlsanity.com/paper/2511.06055/) (2025)
- [FastCar: Cache Attentive Replay for Fast Auto-Regressive Video Generation on the Edge](https://mlsanity.com/paper/2505.14709/) (2025)
- [TrajFlow: Multi-modal Motion Prediction via Flow Matching](https://mlsanity.com/paper/2506.08541/) (2025)
- [Digital Twin Generation from Visual Data: A Survey](https://mlsanity.com/paper/2504.13159/) (2025)
- [ReHyAt: Recurrent Hybrid Attention for Video Diffusion Transformers](https://mlsanity.com/paper/2601.04342/) (2026)
- [Evolution of Video Generative Foundations](https://mlsanity.com/paper/2604.06339/) (2026)
- [Learning What to Trust: Bayesian Prior-Guided Optimization for Visual Generation](https://mlsanity.com/paper/2511.18919/) (2025)
- [Captain Safari: A World Engine with Pose-Aligned 3D Memory](https://mlsanity.com/paper/2511.22815/) (2025)
- [Diverse Video Generation with Determinantal Point Process-Guided Policy Optimization](https://mlsanity.com/paper/2511.20647/) (2025)
- [A Survey: Spatiotemporal Consistency in Video Generation](https://mlsanity.com/paper/2502.17863/) (2025)
- [GenAI Confessions: Black-box Membership Inference for Generative Image Models](https://mlsanity.com/paper/2501.06399/) (2025)
- [Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2510.10671/) (2025)
- [Pre-training for Action Recognition with Automatically Generated Fractal Datasets](https://mlsanity.com/paper/2411.17584/) (2024)
- [Image Generation as a Visual Planner for Robotic Manipulation](https://mlsanity.com/paper/2512.00532/) (2025)
- [ChopGrad: Pixel-Wise Losses for Latent Video Diffusion via Truncated Backpropagation](https://mlsanity.com/paper/2603.17812/) (2026)
- [Control-DINO: Feature Space Conditioning for Controllable Image-to-Video Diffusion](https://mlsanity.com/paper/2604.01761/) (2026)
- [3D-Belief: Embodied Belief Inference via Generative 3D World Modeling](https://mlsanity.com/paper/2605.11367/) (2026)
- [Trading Devil Final: Backdoor attack via Stock market and Bayesian Optimization](https://mlsanity.com/paper/2407.14573/) (2024)
- [Joint Geometry-Appearance Human Reconstruction in a Unified Latent Space via Bridge Diffusion](https://mlsanity.com/paper/2601.00328/) (2026)
- [Reinforcement Learning from Cross-domain Videos with Video Prediction Model](https://mlsanity.com/paper/2606.03201/) (2026)
- [Risk-Averse Inference-Time Diffusion Model Alignment with Finite-Sample Guarantees](https://mlsanity.com/paper/2610.17579/) (2026)
- [Hierarchical Curriculum Learning for Surgical Video Generation: Focusing on Bilateral Axillo-Breast Approach Robotic Thyroidectomy Datasets](https://mlsanity.com/paper/2610.17684/) (2026)

## Abstract (arXiv)

Diffusion generative models have recently become a powerful technique for creating and modifying high-quality, coherent video content. This survey provides a comprehensive overview of the critical components of diffusion models for video generation, including their applications, architectural design, and temporal dynamics modeling. The paper begins by discussing the core principles and mathematical formulations, then explores various architectural choices and methods for maintaining temporal consistency. A taxonomy of applications is presented, categorizing models based on input modalities such as text prompts, images, videos, and audio signals. Advancements in text-to-video generation are discussed to illustrate the state-of-the-art capabilities and limitations of current approaches. Additionally, the survey summarizes recent developments in training and evaluation practices, including the use of diverse video and image datasets and the adoption of various evaluation metrics to assess model performance. The survey concludes with an examination of ongoing challenges, such as generating longer videos and managing computational costs, and offers insights into potential future directions for the field. By consolidating the latest research and developments, this survey aims to serve as a valuable resource for researchers and practitioners working with video diffusion models. Website: https://github.com/ndrwmlnk/Awesome-Video-Diffusion-Models