mlsanityData to 7 Oct 2026

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

arXiv 1906.03327 ↗First posted 7 Jun 2019cs.CVcited by 946

Not on the map. Why?

Cited by 94 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Learning text-video embeddings usually requires a dataset of video clips with manually provided captions. However, such datasets are expensive and time consuming to create and therefore difficult to obtain on a large scale. In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations. The contributions of this work are three-fold. First, we introduce HowTo100M: a large-scale dataset of 136 million video clips sourced from 1.22M narrated instructional web videos depicting humans performing and describing over 23k different visual tasks. Our data collection procedure is fast, scalable and does not require any additional manual annotation. Second, we demonstrate that a text-video embedding trained on this data leads to state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask. Finally, we show that this embedding transfers well to other domains: fine-tuning on generic Youtube videos (MSR-VTT dataset) and movies (LSMDC dataset) outperforms models trained on these datasets alone. Our dataset, code and models will be publicly available at: www.di.ens.fr/willow/research/howto100m/.

Key facts. Cited by 946 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 117; 2022: 188; 2023: 197; 2024: 190; 2025: 164; 2026: 90.

946 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  6. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  7. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  8. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  9. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  10. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  11. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  12. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
38 more
  1. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  2. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  3. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  5. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  6. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  7. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  8. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  9. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  10. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  11. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  12. 2023Learning Interactive Real-World Simulatorscited by 440
  13. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  14. 2022Multimodal Learning with Transformers: A Surveycited by 399
  15. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  16. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  17. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  18. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  19. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  20. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  21. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  22. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  23. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  24. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  25. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  26. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  27. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  28. 2022Egocentric Video-Language Pretrainingcited by 231
  29. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  30. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  31. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  32. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  33. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  34. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 219
  35. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  36. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  37. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  38. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205

50 most cited of 946 All 946 (JSON)