mlsanityData to 14 Oct 2026

Phenaki: Variable Length Video Generation From Open Domain Textual Description

arXiv 2210.02399 ↗First posted 5 Oct 2022cs.CV cs.AIcited by 484

Not on the map. Why?

Cited by 184 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Phenaki, a model capable of realistic video synthesis, given a sequence of textual prompts. Generating videos from text is particularly challenging due to the computational cost, limited quantities of high quality text-video data and variable length of videos. To address these issues, we introduce a new model for learning video representation which compresses the video to a small representation of discrete tokens. This tokenizer uses causal attention in time, which allows it to work with variable-length videos. To generate video tokens from text we are using a bidirectional masked transformer conditioned on pre-computed text tokens. The generated video tokens are subsequently de-tokenized to create the actual video. To address data issues, we demonstrate how joint training on a large corpus of image-text pairs as well as a smaller number of video-text examples can result in generalization beyond what is available in the video datasets. Compared to the previous video generation methods, Phenaki can generate arbitrary long videos conditioned on a sequence of prompts (i.e. time variable text or a story) in open domain. To the best of our knowledge, this is the first time a paper studies generating videos from time variable prompts. In addition, compared to the per-frame baselines, the proposed video encoder-decoder computes fewer tokens per video but results in better spatio-temporal consistency.

Key facts. Cited by 484 papers in the map (data through 14 Oct 2026). Builds on 54 of its 62 references that are in the map. First posted on arXiv 5 Oct 2022.

Builds on

54 of 62 references.

References · 54

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  5. 2022Classifier-Free Diffusion Guidancecited by 5,847
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  12. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
38 more
  1. 2022Video Diffusion Modelscited by 2,305
  2. 2021ViViT: A Video Vision Transformercited by 1,620
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  5. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  6. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  7. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  8. 2021Scaling Vision Transformerscited by 999
  9. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  10. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  11. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  12. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  13. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  14. 2016Generating Videos with Scene Dynamicscited by 376
  15. 2018A Short Note about Kinetics-600cited by 333
  16. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 309
  17. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  18. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  19. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  20. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  21. 2018Stochastic Video Generation with a Learned Priorcited by 245
  22. 2022Diffusion Probabilistic Modeling for Video Generationcited by 238
  23. 2017Stochastic Variational Video Predictioncited by 234
  24. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  25. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  26. 2019Scaling Autoregressive Video Modelscited by 190
  27. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  28. 2017Video Generation From Textcited by 154
  29. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 139
  30. 2022Diffusion Models for Video Prediction and Infillingcited by 135
  31. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  32. 2020Latent Video Transformercited by 106
  33. 2014Video (language) modeling: a baseline for generative models of natural videoscited by 92
  34. 2019High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networkscited by 88
  35. 2021FitVid: Overfitting in Pixel-Level Video Predictioncited by 87
  36. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  37. 2022NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesiscited by 71
  38. 2021CCVS: Context-aware Controllable Video Synthesiscited by 69

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 11; 2023: 135; 2024: 147; 2025: 113; 2026: 78.

484 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  5. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  6. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  7. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  8. 2024Genie: Generative Interactive Environmentscited by 747
  9. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  10. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  11. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  12. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
38 more
  1. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  2. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  3. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  4. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  5. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  6. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  7. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  8. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  9. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
  10. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
  11. 2023MusicLM: Generating Music From Textcited by 448
  12. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 440
  13. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  14. 2022MAGVIT: Masked Generative Video Transformercited by 346
  15. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 327
  16. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 327
  17. 2024Latent Action Pretraining from Videoscited by 321
  18. 2023Pix2Video: Video Editing using Image Diffusioncited by 320
  19. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 314
  20. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 306
  21. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  22. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  23. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 285
  24. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  25. 2025LongLive: Real-time Interactive Long Video Generationcited by 265
  26. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 264
  27. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  28. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 251
  29. 2023Self-Consuming Generative Models Go MADcited by 250
  30. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 246
  31. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  32. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 233
  33. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 228
  34. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
  35. 2023A Survey on Video Diffusion Modelscited by 206
  36. 2023Text-To-4D Dynamic Scene Generationcited by 205
  37. 2025Unified Video Action Modelcited by 205
  38. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 199

50 most cited of 484 All 484 (JSON)