mlsanityData to 14 Oct 2026

ModelScope Text-to-Video Technical Report

arXiv 2308.06571 ↗First posted 12 Aug 2023cs.CV cs.AIcited by 658

On the map: Diffusion models → Video generation

Abstract (arXiv)

This paper introduces ModelScopeT2V, a text-to-video synthesis model that evolves from a text-to-image synthesis model (i.e., Stable Diffusion). ModelScopeT2V incorporates spatio-temporal blocks to ensure consistent frame generation and smooth movement transitions. The model could adapt to varying frame numbers during training and inference, rendering it suitable for both image-text and video-text datasets. ModelScopeT2V brings together three components (i.e., VQGAN, a text encoder, and a denoising UNet), totally comprising 1.7 billion parameters, in which 0.5 billion parameters are dedicated to temporal capabilities. The model demonstrates superior performance over state-of-the-art methods across three evaluation metrics. The code and an online demo are available at urlhttps://modelscope.cn/models/damo/text-to-video-synthesis/summary.

Key facts. Cited by 658 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 62 of its 69 references that are in the map. First posted on arXiv 12 Aug 2023.

Builds on

62 of 69 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  2. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  3. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  4. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  6. 2022Diffusion Models for Video Prediction and Infilling15 Jun
  7. 2022Flexible Diffusion Modeling of Long Videos23 May
  8. 2022Video Diffusion Models7 Apr
  9. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2020Denoising Diffusion Implicit Modelscited by 9,781
  11. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  12. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
38 more
  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2022Classifier-Free Diffusion Guidancecited by 5,847
  5. 2022Generative Adversarial Networkscited by 5,627
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2016Learning without Forgettingcited by 2,496
  10. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  12. 2021ViViT: A Video Vision Transformercited by 1,620
  13. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  14. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,476
  15. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  16. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  17. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  18. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  19. 2021Grounded Language-Image Pre-trainingcited by 1,156
  20. 2021Variational Diffusion Modelscited by 1,153
  21. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  22. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  23. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  24. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 700
  25. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  26. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 532
  27. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  28. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 387
  29. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
  30. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 301
  31. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  32. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  33. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  34. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  35. 2022Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Qualitycited by 182
  36. 2022Riemannian Diffusion Modelscited by 122
  37. 2022Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillationcited by 72
  38. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 55

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 57; 2024: 239; 2025: 240; 2026: 122.

658 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  4. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  5. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  6. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  7. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  8. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  9. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  10. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  11. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  12. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
38 more
  1. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  2. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  3. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 366
  4. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 350
  5. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  6. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  7. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 341
  8. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  9. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  10. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 302
  11. 2024Timestep Embedding Tells: It's Time to Cache for Video Diffusion Modelcited by 269
  12. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  13. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 251
  14. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  15. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 247
  16. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
  17. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  18. 2023A Survey on Video Diffusion Modelscited by 206
  19. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 197
  20. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 188
  21. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 185
  22. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
  23. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Reschedulingcited by 180
  24. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 175
  25. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 174
  26. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 173
  27. 2025History-Guided Video Diffusioncited by 171
  28. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 168
  29. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 168
  30. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  31. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 156
  32. 2025Diffusion Adversarial Post-Training for One-Step Video Generationcited by 151
  33. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 149
  34. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  35. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  36. 2024FIFO-Diffusion: Generating Infinite Videos from Text without Trainingcited by 137
  37. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 135
  38. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 132

50 most cited of 658 All 658 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide
  2. 2025STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolution
  3. 2024Searching Priors Makes Text-to-Video Synthesis Better
  4. 2023FlowZero: Zero-Shot Text-to-Video Synthesis with LLM-Driven Dynamic Scene Syntax
  5. 2024AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction
  6. 2025Video Text Preservation with Synthetic Text-Rich Videos
  7. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
  8. 2025VideoVerse: Does Your T2V Generator Have World Model Capability to Synthesize Videos?