mlsanityData to 14 Oct 2026

VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models

arXiv 2403.06098 ↗First posted 10 Mar 2024cs.CV cs.CLcited by 131

On the map: Diffusion models → Video generation

Abstract (arXiv)

The arrival of Sora marks a new era for text-to-video diffusion models, bringing significant advancements in video generation and potential applications. However, Sora, along with other text-to-video diffusion models, is highly reliant on prompts, and there is no publicly available dataset that features a study of text-to-video prompts. In this paper, we introduce VidProM, the first large-scale dataset comprising 1.67 Million unique text-to-Video Prompts from real users. Additionally, this dataset includes 6.69 million videos generated by four state-of-the-art diffusion models, alongside some related data. We initially discuss the curation of this large-scale dataset, a process that is both time-consuming and costly. Subsequently, we underscore the need for a new prompt dataset specifically designed for text-to-video generation by illustrating how VidProM differs from DiffusionDB, a large-scale prompt-gallery dataset for image generation. Our extensive and diverse dataset also opens up many exciting new research areas. For instance, we suggest exploring text-to-video prompt engineering, efficient video generation, and video copy detection for diffusion models to develop better, more efficient, and safer models. The project (including the collected dataset VidProM and related code) is publicly available at https://vidprom.github.io under the CC-BY-NC 4.0 License.

Key facts. Cited by 131 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 51 of its 76 references that are in the map. First posted on arXiv 10 Mar 2024.

Builds on

51 of 76 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  2. 2023ModelScope Text-to-Video Technical Report12 Aug
  3. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar

Other references · 48

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2017Billion-scale similarity search with GPUscited by 2,497
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  6. 2019CNN-generated images are surprisingly easy to spot... for nowcited by 847
  7. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  8. 2020Leveraging Frequency Analysis for Deep Fake Image Recognitioncited by 504
  9. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  10. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  11. 2023Towards Universal Fake Image Detectors that Generalize Across Generative Modelscited by 490
  12. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
36 more
  1. 2023DIRE for Diffusion-Generated Image Detectioncited by 390
  2. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 379
  3. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  4. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  5. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  6. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  7. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  8. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  9. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 225
  10. 2020Global Texture Enhancement for Fake Face Detection in the Wildcited by 220
  11. 2020Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoningcited by 180
  12. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 154
  13. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  14. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  15. 2023FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generationcited by 122
  16. 2020Multi-modal Dense Video Captioningcited by 113
  17. 2023CelebV-Text: A Large-Scale Facial Text-Video Datasetcited by 104
  18. 2018Reconstruction Network for Video Captioningcited by 98
  19. 2022Fusing Global and Local Features for Generalized AI-Synthesized Image Detectioncited by 88
  20. 2023AntifakePrompt: Prompt-Tuned Vision-Language Models are Fake Image Detectorscited by 64
  21. 2022Investigating Prompt Engineering in Diffusion Modelscited by 58
  22. 2024Towards A Better Metric for Text-to-Video Generationcited by 51
  23. 2023Exploring Effective Factors for Improving Visual In-Context Learningcited by 43
  24. 2021Civil Rephrases Of Toxic Texts With Self-Supervised Transformerscited by 32
  25. 2023WizMap: Scalable Interactive Visualization for Exploring Large Machine Learning Embeddingscited by 28
  26. 2024Frequency Masking for Universal Deepfake Detectioncited by 23
  27. 2023Multi-event Video-Text Retrievalcited by 15
  28. 2021Bag of Tricks and A Strong baseline for Image Copy Detectioncited by 14
  29. 2022Results and findings of the 2021 Image Similarity Challengecited by 10
  30. 2023The 2023 Video Similarity Dataset and Challengecited by 9
  31. 2022So Different Yet So Alike! Constrained Unsupervised Text Style Transfercited by 7
  32. 2024Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineeringcited by 6
  33. 2023A Dual-level Detection Method for Video Copy Detectioncited by 3
  34. 2023A Similarity Alignment Model for Video Copy Segment Matchingcited by 3
  35. 2023Feature-compatible Progressive Learning for Video Copy Detectioncited by 2
  36. 20233rd Place Solution to Meta AI Video Similarity Challengecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 22; 2025: 40; 2026: 69.

131 papers.

Sort:

By area:

  1. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  2. 2025DanceGRPO: Unleashing GRPO on Visual Generationcited by 413
  3. 2025LongLive: Real-time Interactive Long Video Generationcited by 265
  4. 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 234
  5. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 197
  6. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 192
  7. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
  8. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 184
  9. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 156
  10. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 108
  11. 2025SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformercited by 94
  12. 2025Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillationcited by 86
38 more
  1. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 76
  2. 2025OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generationcited by 73
  3. 2024Evaluation of Text-to-Video Generation Models: A Dynamics Perspectivecited by 67
  4. 2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
  5. 2026Helios: Real Real-Time Long Video Generation Modelcited by 65
  6. 2026Context Forcing: Consistent Autoregressive Video Generation with Long Contextcited by 59
  7. 2025STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolutioncited by 56
  8. 2026Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generationcited by 56
  9. 2025VMBench: A Benchmark for Perception-Aligned Video Motion Generationcited by 49
  10. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 48
  11. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  12. 2025VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memorycited by 42
  13. 2025VideoScore2: Think before You Score in Generative Video Evaluationcited by 40
  14. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 39
  15. 2024SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Datasetcited by 35
  16. 2025AI-Generated Video Detection via Perceptual Straighteningcited by 34
  17. 2025D3: Training-Free AI-Generated Video Detection Using Second-Order Featurescited by 34
  18. 2024Video Diffusion Models: A Surveycited by 30
  19. 2025VPO: Aligning Text-to-Video Generation Models with Prompt Optimizationcited by 30
  20. 2026Transition Matching Distillation for Fast Video Generationcited by 30
  21. 2025T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generationcited by 24
  22. 2025NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generationcited by 22
  23. 2026RealWonder: Real-Time Physical Action-Conditioned Video Generationcited by 22
  24. 2024Distinguish Any Fake Videos: Unleashing the Power of Large-scale Data and Motion Featurescited by 21
  25. 2024SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformerscited by 21
  26. 2025VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generationcited by 21
  27. 2025T2V-OptJail: Discrete Prompt Optimization for Text-to-Video Jailbreak Attackscited by 21
  28. 2025T2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generationcited by 19
  29. 2026Streaming Autoregressive Video Generation via Diagonal Distillationcited by 18
  30. 2024Sakuga-42M Dataset: Scaling Up Cartoon Researchcited by 17
  31. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 16
  32. 2026Anchor Forcing: Anchor Memory and Tri-Region RoPE for Interactive Streaming Video Diffusioncited by 16
  33. 2024Is this Generated Person Existed in Real-world? Fine-grained Detecting and Calibrating Abnormal Human-bodycited by 12
  34. 2025Jailbreaking on Text-to-Video Models via Scene Splitting Strategycited by 12
  35. 2024StainDiffuser: MultiTask Dual Diffusion Model for Virtual Stainingcited by 11
  36. 2025Safe-Sora: Safe Text-to-Video Generation via Graphical Watermarkingcited by 11
  37. 2026Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generationcited by 11
  38. 2024A Survey on Multimodal Wearable Sensor-based Human Action Recognitioncited by 10

50 most cited of 131 All 131 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLM
  2. 2025ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models
  3. 2024VideoTetris: Towards Compositional Text-to-Video Generation
  4. 2023VideoBooth: Diffusion-based Video Generation with Image Prompts
  5. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applications
  6. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers
  7. 2024Investigating Memorization in Video Diffusion Models
  8. 2024Long Video Diffusion Generation with Segmented Cross-Attention and Content-Rich Video Data Curation