# Mining YouTube - A dataset for learning fine-grained action concepts from webly supervised video data

arXiv: https://arxiv.org/abs/1906.01012 · HTML: https://mlsanity.com/paper/1906.01012/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2019.

## Built on by (most cited first, 9 of 9)

- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval](https://mlsanity.com/paper/2112.04446/) (2021)
- [Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos](https://mlsanity.com/paper/2104.12671/) (2021)
- [What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions](https://mlsanity.com/paper/2303.16990/) (2023)
- [Routing with Self-Attention for Multimodal Capsule Networks](https://mlsanity.com/paper/2112.00775/) (2021)
- [CAPro: Webly Supervised Learning with Cross-Modality Aligned Prototypes](https://mlsanity.com/paper/2310.09761/) (2023)
- [Implicit Affordance Acquisition via Causal Action-Effect Modeling in the Video Domain](https://mlsanity.com/paper/2312.11345/) (2023)
- [Learning from Weakly-labeled Web Videos via Exploring Sub-Concepts](https://mlsanity.com/paper/2101.03713/) (2021)
- [VideoGEM: Training-free Action Grounding in Videos](https://mlsanity.com/paper/2503.20348/) (2025)

## Abstract (arXiv)

Action recognition is so far mainly focusing on the problem of classification of hand selected preclipped actions and reaching impressive results in this field. But with the performance even ceiling on current datasets, it also appears that the next steps in the field will have to go beyond this fully supervised classification. One way to overcome those problems is to move towards less restricted scenarios. In this context we present a large-scale real-world dataset designed to evaluate learning techniques for human action recognition beyond hand-crafted datasets. To this end we put the process of collecting data on its feet again and start with the annotation of a test set of 250 cooking videos. The training data is then gathered by searching for the respective annotated classes within the subtitles of freely available videos. The uniqueness of the dataset is attributed to the fact that the whole process of collecting the data and training does not involve any human intervention. To address the problem of semantic inconsistencies that arise with this kind of training data, we further propose a semantical hierarchical structure for the mined classes.