mlsanityData to 7 Oct 2026

Latte: Latent Diffusion Transformer for Video Generation

arXiv 2401.03048 ↗First posted 5 Jan 2024cs.CVcited by 536

On the map: Diffusion models → Video generation

Abstract (arXiv)

We propose Latte, a novel Latent Diffusion Transformer for video generation. Latte first extracts spatio-temporal tokens from input videos and then adopts a series of Transformer blocks to model video distribution in the latent space. In order to model a substantial number of tokens extracted from videos, four efficient variants are introduced from the perspective of decomposing the spatial and temporal dimensions of input videos. To improve the quality of generated videos, we determine the best practices of Latte through rigorous experimental analysis, including video clip patch embedding, model variants, timestep-class information injection, temporal positional embedding, and learning strategies. Our comprehensive evaluation demonstrates that Latte achieves state-of-the-art performance across four standard video generation datasets, i.e., FaceForensics, SkyTimelapse, UCF101, and Taichi-HD. In addition, we extend Latte to the text-to-video generation (T2V) task, where Latte achieves results that are competitive with recent T2V models. We strongly believe that Latte provides valuable insights for future research on incorporating Transformers into diffusion models for video generation.

Key facts. Cited by 536 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 72 of its 95 references that are in the map. First posted on arXiv 5 Jan 2024.

Builds on

72 of 95 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Photorealistic Video Generation with Diffusion Models11 Dec
  2. 2023Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation7 Dec
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  4. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction31 Oct
  5. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modeling22 May
  6. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  7. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  8. 2023Video Probabilistic Diffusion Models in Projected Latent Space15 Feb
  9. 2022VIDM: Video Implicit Diffusion Models1 Dec
  10. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation23 Nov
  11. 2022Flexible Diffusion Modeling of Long Videos23 May
  12. 2022Video Diffusion Models7 Apr

Other references · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2020Scaling Laws for Neural Language Modelscited by 6,411
38 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  9. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  10. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  11. 2021ViViT: A Video Vision Transformercited by 1,610
  12. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  13. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  14. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  15. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  16. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  17. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  18. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  19. 2018Image Transformercited by 770
  20. 2020First Order Motion Model for Image Animationcited by 698
  21. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  22. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  23. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  24. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  25. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  26. 2023Zero-shot Image-to-Image Translationcited by 503
  27. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  28. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  29. 2016Generating Videos with Scene Dynamicscited by 376
  30. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  31. 20223D Neural Field Generation using Triplane Diffusioncited by 274
  32. 2021Video Transformer Networkcited by 271
  33. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  34. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  35. 2019Scaling Autoregressive Video Modelscited by 189
  36. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  37. 2022EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equationscited by 172
  38. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 163; 2025: 229; 2026: 144.

536 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  2. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  3. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  4. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  5. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  6. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  7. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  8. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  9. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  10. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  11. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 262
  12. 2024Timestep Embedding Tells: It's Time to Cache for Video Diffusion Modelcited by 245
38 more
  1. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
  2. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 196
  3. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  4. 2025History-Guided Video Diffusioncited by 165
  5. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 163
  6. 2025Open-Sora 2.0: Training a Commercial-Level Video Generation Model in 200kcited by 157
  7. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  8. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  9. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 152
  10. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  11. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  12. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 140
  13. 2024FIFO-Diffusion: Generating Infinite Videos from Text without Trainingcited by 134
  14. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  15. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 124
  16. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  17. 2025Fast Video Generation with Sliding Tile Attentioncited by 123
  18. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  19. 2024OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generationcited by 117
  20. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 115
  21. 2024FasterCache: Training-Free Video Diffusion Model Acceleration with High Qualitycited by 115
  22. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 114
  23. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  24. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 110
  25. 2024UniScene: Unified Occupancy-centric Driving Scene Generationcited by 102
  26. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 100
  27. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 100
  28. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 97
  29. 2024IRASim: A Fine-Grained World Model for Robot Manipulationcited by 93
  30. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 92
  31. 2024SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpointscited by 92
  32. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  33. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 85
  34. 2024StableAnimator: High-Quality Identity-Preserving Human Image Animationcited by 85
  35. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 76
  36. 2024EasyAnimate: High-Performance Video Generation Framework with Hybrid Windows Attention and Reward Backpropagationcited by 76
  37. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74
  38. 2025OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generationcited by 73

50 most cited of 536 All 536 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Rate
  2. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models
  3. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Models
  4. 2024S2DM: Sector-Shaped Diffusion Models for Video Generation
  5. 2024Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition
  6. 2024LTX-Video: Realtime Video Latent Diffusion
  7. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers
  8. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation