mlsanityData to 7 Oct 2026

Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content

arXiv 2410.08260 ↗First posted 10 Oct 2024cs.CV cs.AIcited by 132

Not on the map. Why?

Cited by 32 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

With the continuous progress of visual generation technologies, the scale of video datasets has grown exponentially. The quality of these datasets plays a pivotal role in the performance of video generation models. We assert that temporal splitting, detailed captions, and video quality filtering are three crucial determinants of dataset quality. However, existing datasets exhibit various limitations in these areas. To address these challenges, we introduce Koala-36M, a large-scale, high-quality video dataset featuring accurate temporal splitting, detailed captions, and superior video quality. The essence of our approach lies in improving the consistency between fine-grained conditions and video content. Specifically, we employ a linear classifier on probability distributions to enhance the accuracy of transition detection, ensuring better temporal consistency. We then provide structured captions for the splitted videos, with an average length of 200 words, to improve text-video alignment. Additionally, we develop a Video Training Suitability Score (VTSS) that integrates multiple sub-metrics, allowing us to filter high-quality videos from the original corpus. Finally, we incorporate several metrics into the training process of the generation model, further refining the fine-grained conditions. Our experiments demonstrate the effectiveness of our data processing pipeline and the quality of the proposed Koala-36M dataset. Our dataset and code have been released at https://koala36m.github.io/.

Key facts. Cited by 132 papers in the map (data through 7 Oct 2026). Builds on 36 of its 40 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

36 of 40 references.

References · 36

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  4. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  5. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  6. 2017Localizing Moments in Video with Natural Languagecited by 660
  7. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  8. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  9. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  10. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  11. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  12. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
24 more
  1. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  2. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  3. 2022Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectivescited by 314
  4. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  5. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  6. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  7. 2015A Dataset for Movie Descriptioncited by 195
  8. 2022FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Samplingcited by 179
  9. 2023Video Probabilistic Diffusion Models in Projected Latent Spacecited by 171
  10. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  11. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  12. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  13. 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 140
  14. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  15. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
  16. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 102
  17. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  18. 2023GenTron: Diffusion Transformers for Image and Video Generationcited by 80
  19. 2023D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruningcited by 68
  20. 2024VidGen-1M: A Large-Scale Dataset for Text-to-video Generationcited by 60
  21. 2021ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learningcited by 50
  22. 2023Dataset Quantizationcited by 47
  23. 2023Zoom-VQA: Patches, Frames and Clips Integration for Video Quality Assessmentcited by 21
  24. 2024Enhancing Blind Video Quality Assessment with Rich Quality-aware Featurescited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 81; 2026: 50.

132 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  2. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  3. 2025World Simulation with Video Foundation Models for Physical AIcited by 183
  4. 2026Advancing Open-source World Modelscited by 144
  5. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 120
  6. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  7. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  8. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 88
  9. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 87
  10. 2025HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generationcited by 84
  11. 2024OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generationcited by 77
  12. 2025OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generationcited by 73
38 more
  1. 2025SpatialVID: A Large-Scale Video Dataset with Spatial Annotationscited by 72
  2. 2025HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characterscited by 64
  3. 2025Let Them Talk: Audio-Driven Multi-Person Conversational Video Generationcited by 64
  4. 2025Wan-S2V: Audio-Driven Cinematic Video Generationcited by 60
  5. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 54
  6. 2025UltraVideo: High-Quality UHD Video Dataset with Comprehensive Captionscited by 52
  7. 2025CamCloneMaster: Enabling Reference-based Camera Control for Video Generationcited by 44
  8. 2025HuMo: Human-Centric Video Generation via Collaborative Multi-Modal Conditioningcited by 44
  9. 2025VideoScore2: Think before You Score in Generative Video Evaluationcited by 37
  10. 2026Infinite Worlds with Versatile Interactionscited by 37
  11. 2025VMoBA: Mixture-of-Block Attention for Video Diffusion Modelscited by 35
  12. 2025InstructX: Towards Unified Visual Editing with MLLM Guidancecited by 31
  13. 2025EchoShot: Multi-Shot Portrait Video Generationcited by 28
  14. 2025IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessmentcited by 27
  15. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  16. 2025Streaming Video Instruction Tuningcited by 25
  17. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understandingcited by 24
  18. 2025LongCat-Flash-Omni Technical Reportcited by 23
  19. 2025VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generationcited by 22
  20. 2025OmniVDiff: Omni Controllable Video Diffusion for Generation and Understandingcited by 22
  21. 2025VideoUFO: A Million-Scale User-Focused Dataset for Text-to-Video Generationcited by 20
  22. 2025The Quest for Generalizable Motion Generation: Data, Model, and Evaluationcited by 20
  23. 2025MagicDistillation: Weak-to-Strong Video Distillation for Large-Scale Few-Step Synthesiscited by 18
  24. 2025OmniV2V: Versatile Video Generation and Editing via Dynamic Content Manipulationcited by 18
  25. 2025FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformerscited by 17
  26. 2025Phantom-Data : Towards a General Subject-Consistent Video Generation Datasetcited by 17
  27. 2025PolyVivid: Vivid Multi-Subject Video Generation with Cross-Modal Interaction and Enhancementcited by 16
  28. 2025FreeLong++: Training-Free Long Video Generation via Multi-band SpectralFusioncited by 15
  29. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  30. 2025Vidi: Large Multimodal Models for Video Understanding and Editingcited by 14
  31. 2026NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generationcited by 13
  32. 2025RealCam-Vid: High-resolution Video Dataset with Dynamic Scenes and Metric-scale Camera Movementscited by 12
  33. 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Modelcited by 12
  34. 2025MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agentcited by 11
  35. 2025Uniform Discrete Diffusion with Metric Path for Video Generationcited by 11
  36. 2026SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing modelcited by 11
  37. 2025VidBridge-R1: Bridging QA and Captioning for RL-based Video Understanding Models with Intermediate Proxy Taskscited by 10
  38. 2026Action100M: A Large-scale Video Action Datasetcited by 10

50 most cited of 132 All 132 (JSON)