mlsanityData to 7 Oct 2026

VideoPoet: A Large Language Model for Zero-Shot Video Generation

arXiv 2312.14125 ↗First posted 21 Dec 2023cs.CV cs.AIcited by 479

Not on the map. Why?

Cited by 112 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals. VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio. The training protocol follows that of Large Language Models (LLMs), consisting of two stages: pretraining and task-specific adaptation. During pretraining, VideoPoet incorporates a mixture of multimodal generative objectives within an autoregressive Transformer framework. The pretrained LLM serves as a foundation that can be adapted for a range of video generation tasks. We present empirical results demonstrating the model's state-of-the-art capabilities in zero-shot video generation, specifically highlighting VideoPoet's ability to generate high-fidelity motions. Project page: http://sites.research.google/videopoet/

Key facts. Cited by 479 papers in the map (data through 7 Oct 2026). Builds on 82 of its 97 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

82 of 97 references.

References · 82

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  10. 2022Video Diffusion Modelscited by 2,280
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  13. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  14. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  15. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  16. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  17. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  18. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  19. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  20. 2023PaLM 2 Technical Reportcited by 1,154
  21. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  22. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  23. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  24. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  25. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  26. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  27. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  28. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  29. 2023ModelScope Text-to-Video Technical Reportcited by 651
  30. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  31. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  32. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  33. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  34. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  35. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  36. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  37. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  38. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 163; 2025: 190; 2026: 124.

479 papers.

Sort:

By area:

  1. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  2. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  3. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  4. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  5. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  6. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  7. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  8. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  9. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  10. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  11. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  12. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
38 more
  1. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  2. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  3. 2024Navigation World Modelscited by 262
  4. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 262
  5. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  6. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  7. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  8. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  9. 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 209
  10. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 204
  11. 2023Video Understanding with Large Language Models: A Surveycited by 203
  12. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 197
  13. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  14. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  15. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 185
  16. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  17. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  18. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 169
  19. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 159
  20. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 157
  21. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  22. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  23. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 152
  24. 2025Do generative video models understand physical principles?cited by 151
  25. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters Morecited by 149
  26. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  27. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  28. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  29. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 141
  30. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  31. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  32. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  33. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 125
  34. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 125
  35. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Modelscited by 124
  36. 2025Video World Models with Long-term Spatial Memorycited by 124
  37. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  38. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119

50 most cited of 479 All 479 (JSON)