mlsanityData to 7 Oct 2026

VidGen-1M: A Large-Scale Dataset for Text-to-video Generation

arXiv 2408.02629 ↗First posted 5 Aug 2024cs.CVcited by 60

Not on the map. Why?

Abstract (arXiv)

The quality of video-text pairs fundamentally determines the upper bound of text-to-video models. Currently, the datasets used for training these models suffer from significant shortcomings, including low temporal consistency, poor-quality captions, substandard video quality, and imbalanced data distribution. The prevailing video curation process, which depends on image models for tagging and manual rule-based curation, leads to a high computational load and leaves behind unclean data. As a result, there is a lack of appropriate training datasets for text-to-video models. To address this problem, we present VidGen-1M, a superior training dataset for text-to-video models. Produced through a coarse-to-fine curation strategy, this dataset guarantees high-quality videos and detailed captions with excellent temporal consistency. When used to train the video generation model, this dataset has led to experimental results that surpass those obtained with other models.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 5 Aug 2024.

Builds on

19 of 26 references.

References · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  6. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  7. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  8. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  9. 2017Localizing Moments in Video with Natural Languagecited by 660
  10. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  11. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  12. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
7 more
  1. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  2. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  3. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  4. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  5. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  6. 2015A Dataset for Movie Descriptioncited by 195
  7. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 30; 2026: 24.

60 papers.

Sort:

By area:

  1. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 167
  2. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  3. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  4. 2025SpatialVID: A Large-Scale Video Dataset with Spatial Annotationscited by 72
  5. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Designcited by 54
  6. 2025UltraVideo: High-Quality UHD Video Dataset with Comprehensive Captionscited by 52
  7. 2024LiFT: Leveraging Human Feedback for Text-to-Video Model Alignmentcited by 51
  8. 2024EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generationcited by 38
  9. 2026Infinite Worlds with Versatile Interactionscited by 37
  10. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  11. 2025VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generationcited by 20
  12. 2025BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representationscited by 15
38 more
  1. 2025DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Trainingcited by 15
  2. 2025FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusioncited by 15
  3. 2025Turbo-VAED: Fast and Stable Transfer of Video-VAEs to Mobile Devicescited by 15
  4. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 14
  5. 2025On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devicescited by 14
  6. 2025Think Before You Diffuse: Infusing Physical Rules into Video Diffusioncited by 14
  7. 2025Dual-IPO: Dual-Iterative Preference Optimization for Text-to-Video Generationcited by 12
  8. 2025MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agentcited by 11
  9. 2025Frame In-N-Out: Unbounded Controllable Image-to-Video Generationcited by 10
  10. 2026Action100M: A Large-scale Video Action Datasetcited by 10
  11. 2025TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videoscited by 9
  12. 2025MiCo: Multi-image Contrast for Reinforcement Visual Reasoningcited by 9
  13. 2024Individual Content and Motion Dynamics Preserved Pruning for Video Diffusion Modelscited by 8
  14. 2025VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluationcited by 8
  15. 2026Tele-Omni: a Unified Multimodal Framework for Video Generation and Editingcited by 8
  16. 2026Kairos: A Regret-Aware Native World-Action Model Stack for Physical AIcited by 8
  17. 2025How Confident are Video Models? Empowering Video Models to Express their Uncertaintycited by 6
  18. 2026Toward Native Multimodal Modeling: A Roadmapcited by 6
  19. 2025ViMix-14M: A Curated Multi-Source Video-Text Dataset with Long-Form, High-Quality Captions and Crawl-Free Accesscited by 5
  20. 2026Self-Improving World Modelling with Latent Actionscited by 5
  21. 2026Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generationcited by 5
  22. 2026OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformercited by 5
  23. 2026CineDance: Towards Next-Generation Multi-Shot Long-Form Cinematic Audio-Video Generationcited by 5
  24. 2025On Equivariance and Fast Sampling in Video Diffusion Models Trained with Warped Noisecited by 4
  25. 2025PhyDetEx: Detecting and Explaining the Physical Plausibility of T2V Modelscited by 4
  26. 2026Motion Attribution for Video Generationcited by 4
  27. 2025CI-VID: A Coherent Interleaved Text-Video Datasetcited by 3
  28. 2026Mind the Generative Details: Direct Localized Detail Preference Optimization for Video Diffusion Modelscited by 3
  29. 2025LayerT2V: A Unified Multi-Layer Video Generation Frameworkcited by 2
  30. 2025What Happens Next? Next Scene Prediction with a Unified Video Modelcited by 2
  31. 2026VideoWeave: A Data-Centric Approach for Efficient Video Understandingcited by 2
  32. 2026A2-Edit: Precise Reference-Guided Image Editing of Arbitrary Objects and Ambiguous Maskscited by 2
  33. 2026Sekai2: From World Exploration to Interactive World Modelingcited by 2
  34. 2025MeshFleet: Filtered and Annotated 3D Vehicle Dataset for Domain Specific Generative Modelingcited by 1
  35. 2025Generating, Fast and Slow: Scalable Parallel Video Generation with Video Interface Networkscited by 1
  36. 2025EBind: a practical approach to space bindingcited by 1
  37. 2026PISCES: Annotation-free Text-to-Video Post-Training via Optimal Transport-Aligned Rewardscited by 1
  38. 2026MotionCFG: Boosting Motion Dynamics via Stochastic Concept Perturbationcited by 1

50 most cited of 60 All 60 (JSON)