mlsanityData to 7 Oct 2026

Self-Supervised MultiModal Versatile Networks

arXiv 2006.16228 ↗First posted 29 Jun 2020cs.CVcited by 276

Not on the map. Why?

Abstract (arXiv)

Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: visual, audio and language streams. To this end, we introduce the notion of a multimodal versatile network -- a network that can ingest multiple modalities and whose representations enable downstream tasks in multiple modalities. In particular, we explore how best to combine the modalities, such that fine-grained representations of the visual and audio modalities can be maintained, whilst also integrating text into a common embedding. Driven by versatility, we also introduce a novel process of deflation, so that the networks can be effortlessly applied to the visual data in the form of video or a static image. We demonstrate how such networks trained on large collections of unlabelled video data can be applied on video, video-text, image and audio tasks. Equipped with these representations, we obtain state-of-the-art performance on multiple challenging benchmarks including UCF101, HMDB51, Kinetics600, AudioSet and ESC-50 when compared to previous self-supervised work. Our models are publicly available.

Key facts. Cited by 276 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 69; 2022: 83; 2023: 66; 2024: 37; 2025: 12; 2026: 9.

276 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  5. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  6. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  7. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  8. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  9. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  10. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  11. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  12. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
38 more
  1. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  2. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  3. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  4. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  5. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  6. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  7. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  8. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  9. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  10. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  11. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  12. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  13. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  14. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  15. 2022From data to functa: Your data point is a function and you can treat it like onecited by 183
  16. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 147
  17. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  18. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  19. 2022X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrievalcited by 127
  20. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 122
  21. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  22. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inferencecited by 112
  23. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  24. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  25. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  26. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  27. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  28. 2022Mind Reader: Reconstructing complex images from brain activitiescited by 85
  29. 2021Robust Audio-Visual Instance Discriminationcited by 80
  30. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  31. 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79
  32. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  33. 2023Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lecturescited by 68
  34. 2022Video Transformers: A Surveycited by 63
  35. 2021Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videoscited by 62
  36. 2021PolyViT: Co-training Vision Transformers on Images, Videos and Audiocited by 58
  37. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  38. 2021VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learningcited by 56

50 most cited of 276 All 276 (JSON)