mlsanityData to 14 Oct 2026

Photorealistic Video Generation with Diffusion Models

arXiv 2312.06662 ↗First posted 11 Dec 2023cs.CV cs.AI cs.LGcited by 306

On the map: Diffusion models → Video generation

Abstract (arXiv)

We present W.A.L.T, a transformer-based approach for photorealistic video generation via diffusion modeling. Our approach has two key design decisions. First, we use a causal encoder to jointly compress images and videos within a unified latent space, enabling training and generation across modalities. Second, for memory and training efficiency, we use a window attention architecture tailored for joint spatial and spatiotemporal generative modeling. Taken together these design decisions enable us to achieve state-of-the-art performance on established video (UCF-101 and Kinetics-600) and image (ImageNet) generation benchmarks without using classifier free guidance. Finally, we also train a cascade of three models for the task of text-to-video generation consisting of a base latent video diffusion model, and two video super-resolution diffusion models to generate videos of 512 × 896 resolution at 8 frames per second.

Key facts. Cited by 306 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 75 of its 88 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

75 of 88 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation9 Oct
  2. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modeling22 May
  3. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  5. 2023Video Probabilistic Diffusion Models in Projected Latent Space15 Feb
  6. 2022Scalable Adaptive Computation for Iterative Generation22 Dec
  7. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation23 Nov
  8. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  9. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  10. 2022Flexible Diffusion Modeling of Long Videos23 May
  11. 2022Video Diffusion Models7 Apr

Other references · 64

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  10. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  11. 2020Denoising Diffusion Implicit Modelscited by 9,781
  12. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2022Scalable Diffusion Models with Transformerscited by 6,054
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  12. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  13. 2016Improved Techniques for Training GANscited by 2,970
  14. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  15. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  16. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  17. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  18. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  19. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  20. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  21. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  22. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  23. 2021Scaling Vision Transformerscited by 999
  24. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  25. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  26. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  27. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  28. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  29. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  30. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  31. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  32. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  33. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 422
  34. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 373
  35. 2022MAGVIT: Masked Generative Video Transformercited by 346
  36. 2016A Learned Representation For Artistic Stylecited by 341
  37. 2018A Short Note about Kinetics-600cited by 333
  38. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 310

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 122; 2025: 114; 2026: 69.

306 papers.

Sort:

By area:

  1. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  2. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 737
  3. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
  4. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  5. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  6. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  7. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  8. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  9. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
  10. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  11. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 393
  12. 2024Diffusion Models Are Real-Time Game Enginescited by 296
38 more
  1. 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 234
  2. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 208
  3. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 197
  4. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 191
  5. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 184
  6. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 174
  7. 2025History-Guided Video Diffusioncited by 171
  8. 2026Advancing Open-source World Modelscited by 154
  9. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 146
  10. 2024Consistency Models Made Easycited by 144
  11. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  12. 2025Video World Models with Long-term Spatial Memorycited by 136
  13. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 129
  14. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 124
  15. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  16. 2024Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Modelscited by 114
  17. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  18. 2024Rolling Diffusion Modelscited by 104
  19. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
  20. 2025One-Minute Video Generation with Test-Time Trainingcited by 101
  21. 2024Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Modelscited by 100
  22. 2024MaskBit: Embedding-free Image Generation via Bit Tokenscited by 98
  23. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 98
  24. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 93
  25. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 92
  26. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 89
  27. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  28. 2025Latent Diffusion Model without Variational Autoencodercited by 88
  29. 2025Infinity-RoPE: Action-Controllable Infinite Video Generation Emerges From Autoregressive Self-Rolloutcited by 86
  30. 2025Seaweed-7B: Cost-Effective Training of Video Generation Foundation Modelcited by 82
  31. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  32. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 80
  33. 2025Improving the Diffusability of Autoencoderscited by 80
  34. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 76
  35. 2024Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusioncited by 75
  36. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  37. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 73
  38. 20243DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generationcited by 71

50 most cited of 306 All 306 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation
  2. 2026Efficient Video Diffusion Models: Advancements and Challenges
  3. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models
  4. 2025CascadeV: An Implementation of Wurstchen Architecture for Video Generation
  5. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
  6. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Models
  7. 2025VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Rate
  8. 2025Lumina-Video: Efficient and Flexible Video Generation with Multi-scale Next-DiT