mlsanityData to 14 Oct 2026

End-to-End Learning of Visual Representations from Uncurated Instructional Videos

arXiv 1912.06430 ↗First posted 13 Dec 2019cs.CVcited by 507

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Annotating videos is cumbersome, expensive and not scalable. Yet, many strong video models still rely on manually annotated data. With the recent introduction of the HowTo100M dataset, narrated videos now offer the possibility of learning video representations without manual supervision. In this work we propose a new learning approach, MIL-NCE, capable of addressing misalignments inherent to narrated videos. With this approach we are able to learn strong video representations from scratch, without the need for any manual annotation. We evaluate our representations on a wide range of four downstream tasks over eight datasets: action recognition (HMDB-51, UCF-101, Kinetics-700), text-to-video retrieval (YouCook2, MSR-VTT), action localization (YouTube-8M Segments, CrossTask) and action segmentation (COIN). Our method outperforms all published self-supervised approaches for these tasks as well as several fully supervised baselines.

Key facts. Cited by 507 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 119; 2022: 145; 2023: 118; 2024: 61; 2025: 38; 2026: 26.

507 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  6. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  7. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  8. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  9. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  10. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  11. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  12. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
38 more
  1. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  2. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  3. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  4. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  5. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  7. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  8. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  9. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  10. 2022Multimodal Learning with Transformers: A Surveycited by 400
  11. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  12. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  13. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  14. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  15. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 280
  16. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  17. 2022Egocentric Video-Language Pretrainingcited by 232
  18. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  19. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 225
  20. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
  21. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  22. 2022Learning Video Representations from Large Language Modelscited by 205
  23. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 198
  24. 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 198
  25. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  26. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  27. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  28. 2022Revisiting the "Video" in Video-Language Understandingcited by 169
  29. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 162
  30. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  31. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  32. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  33. 2021TCLR: Temporal Contrastive Learning for Video Representationcited by 136
  34. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  35. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  36. 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 128
  37. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 124
  38. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117

50 most cited of 507 All 507 (JSON)