mlsanityData to 14 Oct 2026

Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer

arXiv 2204.03638 ↗First posted 7 Apr 2022cs.CV cs.AIcited by 244

Not on the map. Why?

Cited by 107 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Videos are created to express emotion, exchange information, and share experiences. Video synthesis has intrigued researchers for a long time. Despite the rapid progress driven by advances in visual synthesis, most existing studies focus on improving the frames' quality and the transitions between them, while little progress has been made in generating longer videos. In this paper, we present a method that builds on 3D-VQGAN and transformers to generate videos with thousands of frames. Our evaluation shows that our model trained on 16-frame video clips from standard benchmarks such as UCF-101, Sky Time-lapse, and Taichi-HD datasets can generate diverse, coherent, and high-quality long videos. We also showcase conditional extensions of our approach for generating meaningful long videos by incorporating temporal information with text and audio. Videos and code can be found at https://songweige.github.io/projects/tats/index.html.

Key facts. Cited by 244 papers in the map (data through 14 Oct 2026). Builds on 71 of its 87 references that are in the map. First posted on arXiv 7 Apr 2022.

Builds on

71 of 87 references.

References · 71

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  8. 2017Neural Discrete Representation Learningcited by 4,712
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  11. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
38 more
  1. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  3. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  4. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  5. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
  6. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  7. 2018Group Normalizationcited by 1,524
  8. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  9. 2018Hierarchical Neural Story Generationcited by 1,193
  10. 2021Alias-Free Generative Adversarial Networkscited by 1,184
  11. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,179
  12. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  13. 2020First Order Motion Model for Image Animationcited by 704
  14. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  15. 2020Jukebox: A Generative Model for Musiccited by 542
  16. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  17. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  18. 2018Video-to-Video Synthesiscited by 390
  19. 2016Generating Videos with Scene Dynamicscited by 376
  20. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  21. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 301
  22. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 264
  23. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  24. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  25. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  26. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  27. 2019Scaling Autoregressive Video Modelscited by 190
  28. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  29. 2020How Much Position Information Do Convolutional Neural Networks Encode?cited by 172
  30. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 163
  31. 2017Video Generation From Textcited by 154
  32. 2020On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Locationcited by 142
  33. 2020Adversarial Generation of Continuous Imagescited by 135
  34. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  35. 2021ViTGAN: Training GANs with Vision Transformerscited by 110
  36. 2016Video Pixel Networkscited by 108
  37. 2020Latent Video Transformercited by 106
  38. 2020Image Generators with Conditionally-Independent Pixel Synthesiscited by 106

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 20; 2023: 67; 2024: 71; 2025: 62; 2026: 24.

244 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  5. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  6. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  7. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  8. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  9. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  10. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  11. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  12. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
38 more
  1. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  2. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  3. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  4. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  5. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
  6. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 388
  7. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  8. 2022MAGVIT: Masked Generative Video Transformercited by 346
  9. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 331
  10. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
  11. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  12. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 302
  13. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  14. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 285
  15. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 233
  16. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 227
  17. 2023A Survey on Video Diffusion Modelscited by 206
  18. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 188
  19. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 185
  20. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Reschedulingcited by 180
  21. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 173
  22. 2023Video Probabilistic Diffusion Models in Projected Latent Spacecited by 171
  23. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generationcited by 171
  24. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 167
  25. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 155
  26. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 149
  27. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 141
  28. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  29. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 139
  30. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generationcited by 137
  31. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 132
  32. 2024ID-Animator: Zero-Shot Identity-Preserving Human Video Generationcited by 124
  33. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 123
  34. 2024OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generationcited by 118
  35. 2022Generating Long Videos of Dynamic Scenescited by 113
  36. 2024Image and Video Tokenization with Binary Spherical Quantizationcited by 112
  37. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  38. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105

50 most cited of 244 All 244 (JSON)