mlsanityData to 14 Oct 2026

Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis

arXiv 2402.14797 ↗First posted 22 Feb 2024cs.CV cs.AIcited by 105

Not on the map. Why?

Cited by 53 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contemporary models for generating images show remarkable quality and versatility. Swayed by these advantages, the research community repurposes them to generate videos. Since video content is highly redundant, we argue that naively bringing advances of image models to the video generation domain reduces motion fidelity, visual quality and impairs scalability. In this work, we build Snap Video, a video-first model that systematically addresses these challenges. To do that, we first extend the EDM framework to take into account spatially and temporally redundant pixels and naturally support video generation. Second, we show that a U-Net - a workhorse behind image generation - scales poorly when generating videos, requiring significant computational overhead. Hence, we propose a new transformer-based architecture that trains 3.31 times faster than U-Nets (and is ~4.5 faster at inference). This allows us to efficiently train a text-to-video model with billions of parameters for the first time, reach state-of-the-art results on a number of benchmarks, and generate videos with substantially higher quality, temporal consistency, and motion complexity. The user studies showed that our model was favored by a large margin over the most recent methods. See our website at https://snap-research.github.io/snapvideo/.

Key facts. Cited by 105 papers in the map (data through 14 Oct 2026). Builds on 66 of its 73 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

66 of 73 references.

References · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
38 more
  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  2. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  3. 2016Improved Techniques for Training GANscited by 2,970
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  5. 2022Video Diffusion Modelscited by 2,305
  6. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  7. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  8. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  9. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  10. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  11. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  12. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  13. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  14. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  15. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  16. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  17. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  18. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  19. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  20. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  21. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
  22. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  23. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  24. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 422
  25. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 388
  26. 2022MAGVIT: Masked Generative Video Transformercited by 346
  27. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  28. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 331
  29. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 301
  30. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 285
  31. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  32. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  33. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  34. 2023SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Secondscited by 227
  35. 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 213
  36. 2023On the Importance of Noise Scheduling for Diffusion Modelscited by 194
  37. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  38. 2018Stochastic Adversarial Video Predictioncited by 173

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 39; 2025: 49; 2026: 17.

105 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  2. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  3. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  4. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 287
  5. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 161
  6. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  7. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 154
  8. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 146
  9. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 99
  10. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 98
  11. 2025Improving the Diffusability of Autoencoderscited by 80
  12. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 77
38 more
  1. 2024ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuningcited by 70
  2. 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Modelcited by 66
  3. 2025Multi-subject Open-set Personalization in Video Generationcited by 66
  4. 20244Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Modelscited by 64
  5. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  6. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49
  7. 2025StreamDiT: Real-Time Streaming Text-to-Video Generationcited by 48
  8. 2025FullDiT: Multi-Task Video Generative Foundation Model with Full Attentioncited by 47
  9. 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 45
  10. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  11. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 44
  12. 2024SF-V: Single Forward Video Generation Modelcited by 40
  13. 2024Mind the Time: Temporally-Controlled Multi-Event Video Generationcited by 40
  14. 2024Video Motion Transfer with Diffusion Transformerscited by 39
  15. 2024BitsFusion: 1.99 bits Weight Quantization of Diffusion Modelcited by 37
  16. 2025Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistencycited by 35
  17. 2025OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulationcited by 32
  18. 2024Taming Data and Transformers for Audio Generationcited by 28
  19. 2025FlowR: Flowing from Sparse to Dense 3D Reconstructionscited by 28
  20. 2024Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversioncited by 27
  21. 20244Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusioncited by 27
  22. 2024SnapGen: Taming High-Resolution Text-to-Image Models for Mobile Devices with Efficient Architectures and Trainingcited by 27
  23. 2024Hierarchical Patch Diffusion Models for High-Resolution Video Generationcited by 24
  24. 2025Generative Video Motion Editing with 3D Point Trackscited by 23
  25. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 23
  26. 2024AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generationcited by 22
  27. 2025Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generationcited by 20
  28. 2025GeoVideo: Introducing Geometric Regularization into Video Generation Modelcited by 19
  29. 2024AKiRa: Augmentation Kit on Rays for optical video generationcited by 18
  30. 2025OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditionscited by 17
  31. 2024Using Diffusion Priors for Video Amodal Segmentationcited by 16
  32. 2024FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidancecited by 15
  33. 2025On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devicescited by 15
  34. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 15
  35. 2025AccVideo: Accelerating Video Diffusion Model with Synthetic Datasetcited by 15
  36. 2025MagicTryOn: Harnessing Diffusion Transformer for Garment-Preserving Video Virtual Try-oncited by 15
  37. 20254Real-Video-V2: Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generationcited by 14
  38. 2025Video-As-Prompt: Unified Semantic Control for Video Generationcited by 14

50 most cited of 105 All 105 (JSON)