mlsanityData to 7 Oct 2026

OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation

arXiv 2407.02371 ↗First posted 2 Jul 2024cs.CVcited by 332

Not on the map. Why?

Cited by 104 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Text-to-video (T2V) generation has recently garnered significant attention thanks to the large multi-modality model Sora. However, T2V generation still faces two important challenges: 1) Lacking a precise open sourced high-quality dataset. The previous popular video datasets, e.g. WebVid-10M and Panda-70M, are either with low quality or too large for most research institutions. Therefore, it is challenging but crucial to collect a precise high-quality text-video pairs for T2V generation. 2) Ignoring to fully utilize textual information. Recent T2V methods have focused on vision transformers, using a simple cross attention module for video generation, which falls short of thoroughly extracting semantic information from text prompt. To address these issues, we introduce OpenVid-1M, a precise high-quality dataset with expressive captions. This open-scenario dataset contains over 1 million text-video pairs, facilitating research on T2V generation. Furthermore, we curate 433K 1080p videos from OpenVid-1M to create OpenVidHD-0.4M, advancing high-definition video generation. Additionally, we propose a novel Multi-modal Video Diffusion Transformer (MVDiT) capable of mining both structure information from visual tokens and semantic information from text tokens. Extensive experiments and ablation studies verify the superiority of OpenVid-1M over previous datasets and the effectiveness of our MVDiT.

Key facts. Cited by 332 papers in the map (data through 7 Oct 2026). Builds on 32 of its 35 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

32 of 35 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  6. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  7. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  9. 2019FaceForensics++: Learning to Detect Manipulated Facial Imagescited by 1,095
  10. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  11. 2020First Order Motion Model for Image Animationcited by 698
  12. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
20 more
  1. 2023ModelScope Text-to-Video Technical Reportcited by 651
  2. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  3. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  4. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  5. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  6. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  7. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  8. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  9. 2022Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectivescited by 314
  10. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  11. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  12. 2022Unifying Flow, Stereo and Depth Estimationcited by 270
  13. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  14. 2022Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentationcited by 177
  15. 2023Video Probabilistic Diffusion Models in Projected Latent Spacecited by 171
  16. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  17. 2023CelebV-Text: A Large-Scale Facial Text-Video Datasetcited by 103
  18. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 102
  19. 2017Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networkscited by 88
  20. 2024MagicTime: Time-lapse Video Generation Models as Metamorphic Simulatorscited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 37; 2025: 150; 2026: 145.

332 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  3. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  4. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  5. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  6. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 143
  7. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusioncited by 139
  8. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  9. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 120
  10. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 116
  11. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  12. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 88
38 more
  1. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 87
  2. 2024OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generationcited by 77
  3. 2025OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generationcited by 73
  4. 2025SpatialVID: A Large-Scale Video Dataset with Spatial Annotationscited by 72
  5. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  6. 2024Allegro: Open the Black Box of Commercial-Level Video Generation Modelcited by 68
  7. 2025Goku: Flow Based Video Generative Foundation Modelscited by 67
  8. 2025Let Them Talk: Audio-Driven Multi-Person Conversational Video Generationcited by 64
  9. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 64
  10. 2025InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Constructioncited by 62
  11. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
  12. 2024Dynamic Diffusion Transformercited by 60
  13. 2025OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modelingcited by 59
  14. 2025STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolutioncited by 56
  15. 2024Exploiting Multimodal Spatial-temporal Patterns for Video Object Trackingcited by 55
  16. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Designcited by 54
  17. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 54
  18. 2025UltraVideo: High-Quality UHD Video Dataset with Comprehensive Captionscited by 52
  19. 2024LiFT: Leveraging Human Feedback for Text-to-Video Model Alignmentcited by 51
  20. 2025OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editingcited by 50
  21. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scalecited by 49
  22. 2024GenXD: Generating Any 3D and 4D Scenescited by 49
  23. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  24. 2024Improved Video VAE for Latent Video Diffusion Modelcited by 46
  25. 2026Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generationcited by 44
  26. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  27. 2025DiP: Taming Diffusion Models in Pixel Spacecited by 42
  28. 2024ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generationcited by 41
  29. 2025Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesiscited by 39
  30. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 39
  31. 2024EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generationcited by 38
  32. 2026Infinite Worlds with Versatile Interactionscited by 37
  33. 2024OSV: One Step is Enough for High-Quality Image to Video Generationcited by 36
  34. 2024MovieBench: A Hierarchical Movie Level Dataset for Long Video Generationcited by 35
  35. 2024AnimateAnything: Consistent and Controllable Animation for Video Generationcited by 34
  36. 2025Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillationcited by 33
  37. 2025EX-4D: EXtreme Viewpoint 4D Video Synthesis via Depth Watertight Meshcited by 32
  38. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32

50 most cited of 332 All 332 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Recipe for Scaling up Text-to-Video Generation with Text-free Videos
  2. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation
  3. 2025VideoVerse: Does Your T2V Generator Have World Model Capability to Synthesize Videos?
  4. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models
  5. 2025The Best of Both Worlds: Integrating Language Models and Diffusion Models for Video Generation
  6. 2024Towards A Better Metric for Text-to-Video Generation
  7. 2025RISE-T2V: Rephrasing and Injecting Semantics with LLM for Expansive Text-to-Video Generation
  8. 2023DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generation