mlsanityData to 7 Oct 2026

Labelling unlabelled videos from scratch with multi-modal self-supervision

arXiv 2006.13662 ↗First posted 24 Jun 2020cs.CV cs.LGcited by 99

Not on the map. Why?

Abstract (arXiv)

A large part of the current success of deep learning lies in the effectiveness of data -- more precisely: labelled data. Yet, labelling a dataset with human annotation continues to carry high costs, especially for videos. While in the image domain, recent methods have allowed to generate meaningful (pseudo-) labels for unlabelled datasets without supervision, this development is missing for the video domain where learning feature representations is the current focus. In this work, we a) show that unsupervised labelling of a video dataset does not come for free from strong feature encoders and b) propose a novel clustering method that allows pseudo-labelling of a video dataset without any human annotations, by leveraging the natural correspondence between the audio and visual modalities. An extensive analysis shows that the resulting clusters have high semantic overlap to ground truth human labels. We further introduce the first benchmarking results on unsupervised labelling of common video datasets Kinetics, Kinetics-Sound, VGG-Sound and AVE.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 26; 2022: 36; 2023: 15; 2024: 18; 2025: 4.

99 papers.

Sort:

By area:

  1. 2021Unsupervised Semantic Segmentation by Contrasting Object Mask Proposalscited by 190
  2. 2022Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detectioncited by 130
  3. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  4. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  5. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 84
  6. 2021Robust Audio-Visual Instance Discriminationcited by 80
  7. 2021Multi-level Feature Learning for Contrastive Multi-view Clusteringcited by 73
  8. 2023Align and Attend: Multimodal Summarization with Dual Contrastive Lossescited by 65
  9. 2021Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videoscited by 62
  10. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  11. 2021Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Datacited by 55
  12. 2022Discovering Object Masks with Transformers for Unsupervised Semantic Segmentationcited by 54
38 more
  1. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  2. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  3. 2021Revisiting Contrastive Methods for Unsupervised Learning of Visual Representationscited by 48
  4. 2022Class-Balanced Pixel-Level Self-Labeling for Domain Adaptive Semantic Segmentationcited by 47
  5. 2022Audiovisual Masked Autoencoderscited by 47
  6. 2022Robust Contrastive Learning against Noisy Viewscited by 45
  7. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  8. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 43
  9. 2024What to align in multimodal contrastive learning?cited by 41
  10. 2021Audio-Visual Synchronisation in the wildcited by 39
  11. 2022ECLIPSE: Efficient Long-range Video Retrieval using Sight and Soundcited by 39
  12. 2023Test-Time Training on Video Streamscited by 37
  13. 2021ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Geneticscited by 35
  14. 2022Audio-Adaptive Activity Recognition Across Video Domainscited by 35
  15. 2021Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learningcited by 34
  16. 2021Multiview Pseudo-Labeling for Semi-supervised Learning from Videocited by 32
  17. 2021Class-aware Sounding Objects Localization via Audiovisual Correspondencecited by 32
  18. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  19. 2022XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learningcited by 30
  20. 2021Self-supervised object detection from audio-visual correspondencecited by 29
  21. 2021Unsupervised Sound Localization via Iterative Contrastive Learningcited by 28
  22. 2021Enhancing Self-supervised Video Representation Learning via Multi-level Feature Optimizationcited by 27
  23. 2022Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Languagecited by 27
  24. 2022Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Datacited by 27
  25. 2022Towards Realistic Semi-Supervised Learningcited by 27
  26. 2021Face, Body, Voice: Video Person-Clustering with Multiple Modalitiescited by 24
  27. 2022How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?cited by 23
  28. 2022Static and Dynamic Concepts for Self-supervised Video Representation Learningcited by 23
  29. 2022Learning State-Aware Visual Representations from Audible Interactionscited by 23
  30. 2022Learning Visual Styles from Audio-Visual Associationscited by 22
  31. 2022Unsupervised Visual Representation Learning by Synchronous Momentum Groupingcited by 22
  32. 2022TVLT: Textless Vision-Language Transformercited by 22
  33. 2022Self-supervised Video Representation Learning with Motion-Aware Masked Autoencoderscited by 20
  34. 2021COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modalitycited by 19
  35. 2022Temporal and cross-modal attention for audio-visual zero-shot learningcited by 19
  36. 2023Hyperbolic Audio-visual Zero-shot Learningcited by 19
  37. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  38. 2021Motion-Augmented Self-Training for Video Recognition at Smaller Scalecited by 17

50 most cited of 99 All 99 (JSON)