mlsanityData to 7 Oct 2026

YouTube-8M: A Large-Scale Video Classification Benchmark

arXiv 1609.08675 ↗First posted 27 Sep 2016cs.CVcited by 324

Not on the map. Why?

Abstract (arXiv)

Many recent advancements in Computer Vision are attributed to large datasets. Open-source software packages for Machine Learning and inexpensive commodity hardware have reduced the barrier of entry for exploring novel approaches at scale. It is possible to train models over millions of examples within a few days. Although large-scale datasets exist for image understanding, such as ImageNet, there are no comparable size video classification datasets. In this paper, we introduce YouTube-8M, the largest multi-label video classification dataset, composed of ~8 million videos (500K hours of video), annotated with a vocabulary of 4800 visual entities. To get the videos and their labels, we used a YouTube video annotation system, which labels videos with their main topics. While the labels are machine-generated, they have high-precision and are derived from a variety of human-based signals including metadata and query click signals. We filtered the video labels (Knowledge Graph entities) using both automated and manual curation strategies, including asking human raters if the labels are visually recognizable. Then, we decoded each video at one-frame-per-second, and used a Deep CNN pre-trained on ImageNet to extract the hidden representation immediately prior to the classification layer. Finally, we compressed the frame features and make both the features and video-level labels available for download. We trained various (modest) classification models on the dataset, evaluated them using popular evaluation metrics, and report them as baselines. Despite the size of the dataset, some of our models train to convergence in less than a day on a single machine using TensorFlow. We plan to release code for training a TensorFlow model and for computing metrics.

Key facts. Cited by 324 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Sep 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 63; 2022: 74; 2023: 46; 2024: 58; 2025: 52; 2026: 31.

324 papers.

Sort:

By area:

  1. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  3. 2023MusicLM: Generating Music From Textcited by 442
  4. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  5. 2022MAGVIT: Masked Generative Video Transformercited by 341
  6. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  7. 2021Dataset Condensation with Differentiable Siamese Augmentationcited by 290
  8. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  9. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  10. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  11. 2023Video Understanding with Large Language Models: A Surveycited by 203
  12. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
38 more
  1. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 189
  2. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  3. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  4. 2021FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Datasetcited by 179
  5. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  6. 2021Occluded Video Instance Segmentation: A Benchmarkcited by 154
  7. 2021TCLR: Temporal Contrastive Learning for Video Representationcited by 136
  8. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  9. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  10. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  11. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
  12. 2022MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Trainingcited by 94
  13. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  14. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  15. 2021SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarioscited by 83
  16. 2021Deepfakes Generation and Detection: State-of-the-art, open challenges, countermeasures, and way forwardcited by 82
  17. 2021MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actionscited by 82
  18. 2022A Survey on Long-Tailed Visual Recognitioncited by 75
  19. 2023YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpuscited by 74
  20. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 73
  21. 2023The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluationcited by 68
  22. 2021Adaptive Methods for Real-World Domain Generalizationcited by 66
  23. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  24. 2021Self-supervised Video Transformercited by 64
  25. 2024Embodied Understanding of Driving Scenarioscited by 63
  26. 2021FineAction: A Fine-Grained Video Dataset for Temporal Action Localizationcited by 60
  27. 2022A CLIP-Hitchhiker's Guide to Long Video Retrievalcited by 57
  28. 2023SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentariescited by 57
  29. 2021Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptionscited by 53
  30. 2024A Survey on Deep Active Learning: Recent Advances and New Frontierscited by 53
  31. 2021ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learningcited by 50
  32. 2022NIRVANA: Neural Implicit Representations of Videos with Adaptive Networks and Autoregressive Patch-wise Modelingcited by 50
  33. 2021SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Imagescited by 49
  34. 2024Pixel-Wise Recognition for Holistic Surgical Scene Understandingcited by 48
  35. 2021Act the Part: Learning Interaction Strategies for Articulated Object Part Discoverycited by 45
  36. 2021Motion-aware Contrastive Video Representation Learning via Foreground-background Mergingcited by 45
  37. 2021Time-Equivariant Contrastive Video Representation Learningcited by 44
  38. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43

50 most cited of 324 All 324 (JSON)