mlsanityData to 14 Oct 2026

VDT: General-purpose Video Diffusion Transformers via Mask Modeling

arXiv 2305.13311 ↗First posted 22 May 2023cs.CVcited by 103

On the map: Diffusion models → Video generation

Abstract (arXiv)

This work introduces Video Diffusion Transformer (VDT), which pioneers the use of transformers in diffusion-based video generation. It features transformer blocks with modularized temporal and spatial attention modules to leverage the rich spatial-temporal representation inherited in transformers. We also propose a unified spatial-temporal mask modeling mechanism, seamlessly integrated with the model, to cater to diverse video generation scenarios. VDT offers several appealing benefits. 1) It excels at capturing temporal dependencies to produce temporally consistent video frames and even simulate the physics and dynamics of 3D objects over time. 2) It facilitates flexible conditioning information, eg, simple concatenation in the token space, effectively unifying different token lengths and modalities. 3) Pairing with our proposed spatial-temporal mask modeling mechanism, it becomes a general-purpose video diffuser for harnessing a range of tasks, including unconditional generation, video prediction, interpolation, animation, and completion, etc. Extensive experiments on these tasks spanning various scenarios, including autonomous driving, natural weather, human action, and physics-based simulation, demonstrate the effectiveness of VDT. Additionally, we present comprehensive studies on how model handles conditioning information with the mask modeling mechanism, which we believe will benefit future research and advance the field. Project page: https:VDT-2023.github.io

Key facts. Cited by 103 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 52 of its 58 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

52 of 58 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  2. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  3. 2023Video Probabilistic Diffusion Models in Projected Latent Space15 Feb
  4. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  5. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  6. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation23 Nov
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  8. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
  9. 2022Video Diffusion Models7 Apr
  10. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 42

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Scalable Diffusion Models with Transformerscited by 6,054
  10. 2022Classifier-Free Diffusion Guidancecited by 5,847
  11. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
30 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  5. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  6. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  7. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 735
  8. 2020First Order Motion Model for Image Animationcited by 704
  9. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 635
  10. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  11. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  12. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  13. 2016Generating Videos with Scene Dynamicscited by 376
  14. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  15. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 311
  16. 2018Stochastic Video Generation with a Learned Priorcited by 245
  17. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  18. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  19. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  20. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 154
  21. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 153
  22. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 147
  23. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  24. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  25. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 114
  26. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  27. 2017Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networkscited by 89
  28. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  29. 2020Learning Long-term Visual Dynamics with Region Proposal Interaction Networkscited by 44
  30. 2022LGDN: Language-Guided Denoising Network for Video-Language Modelingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 41; 2025: 37; 2026: 23.

103 papers.

Sort:

By area:

  1. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  2. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  3. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 393
  4. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 341
  5. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  6. 2024Learning-to-Cache: Accelerating Diffusion Transformer via Layer Cachingcited by 142
  7. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 135
  8. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 108
  9. 2024Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharingcited by 74
  10. 2025SkyReels-A2: Compose Anything in Video Diffusion Transformerscited by 73
  11. 2024Allegro: Open the Black Box of Commercial-Level Video Generation Modelcited by 70
  12. 2024FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modificationcited by 64
38 more
  1. 2024Human4DiT: 360-degree Human Video Generation with 4D Diffusion Transformercited by 61
  2. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 60
  3. 2025StreamDiT: Real-Time Streaming Text-to-Video Generationcited by 48
  4. 2024LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexitycited by 43
  5. 2024Video Motion Transfer with Diffusion Transformerscited by 39
  6. 2024LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformerscited by 37
  7. 2024ViD-GPT: Introducing GPT-style Autoregressive Generation in Video Diffusion Modelscited by 36
  8. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  9. 2024TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Modelcited by 33
  10. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  11. 2024Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Modelcited by 28
  12. 2024TALC: Time-Aligned Captions for Multi-Scene Text-to-Video Generationcited by 25
  13. 2024Diffusion2: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Modelscited by 24
  14. 2024Matten: Video Generation with Mamba-Attentioncited by 24
  15. 2024Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediatorscited by 24
  16. 2025Guiding Human-Object Interactions with Rich Geometry and Relationscited by 24
  17. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 23
  18. 2026Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Modelscited by 22
  19. 2024AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Predictioncited by 21
  20. 2024BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latentscited by 21
  21. 2024SurGen: Text-Guided Diffusion Model for Surgical Video Generationcited by 20
  22. 2024Towards Precise Scaling Laws for Video Diffusion Transformerscited by 20
  23. 2024FrameBridge: Improving Image-to-Video Generation with Bridge Modelscited by 19
  24. 2025FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusioncited by 19
  25. 2025OmniCache: A Trajectory-Oriented Global Perspective on Training-Free Cache Reuse for Diffusion Transformer Modelscited by 18
  26. 2025OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditionscited by 17
  27. 2025RoboEnvision: A Long-Horizon Video Generation Model for Multi-Task Robot Manipulationcited by 16
  28. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 15
  29. 2024Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Modelscited by 14
  30. 2025Ingredients: Blending Custom Photos with Video Diffusion Transformerscited by 13
  31. 2024Video Diffusion Transformers are In-Context Learnerscited by 12
  32. 2025EMMA: Generalizing Real-World Robot Manipulation via Generative Visual Transfercited by 12
  33. 2024GenRec: Unifying Video Generation and Recognition with Diffusion Modelscited by 11
  34. 2024Diffusion Transformers as Open-World Spatiotemporal Foundation Modelscited by 10
  35. 2025Anomaly Detection and Generation with Diffusion Models: A Surveycited by 8
  36. 2025VideoPDE: Unified Generative PDE Solving via Video Inpainting Diffusion Modelscited by 8
  37. 2025CDP: Towards Robust Autoregressive Visuomotor Policy Learning via Causal Diffusioncited by 8
  38. 2024A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approachescited by 7

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023GD-VDM: Generated Depth for better Diffusion-based Video Generation
  2. 2025VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Rate
  3. 2024Anchored Diffusion for Video Face Reenactment
  4. 2026ViFeEdit: A Video-Free Tuner of Your Video Diffusion Transformer
  5. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
  6. 2025DyDiT++: Diffusion Transformers with Timestep and Spatial Dynamics for Efficient Visual Generation
  7. 2024V3D: Video Diffusion Models are Effective 3D Generators
  8. 2025Mask2DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long Video Generation