mlsanityData to 14 Oct 2026

Look, Listen and Learn

arXiv 1705.08168 ↗First posted 23 May 2017cs.CV cs.LGcited by 458

Not on the map. Why?

Abstract (arXiv)

We consider the question: what can be learnt by looking at and listening to a large number of unlabelled videos? There is a valuable, but so far untapped, source of information contained in the video itself -- the correspondence between the visual and the audio streams, and we introduce a novel "Audio-Visual Correspondence" learning task that makes use of this. Training visual and audio networks from scratch, without any additional supervision other than the raw unconstrained videos themselves, is shown to successfully solve this task, and, more interestingly, result in good visual and audio representations. These features set the new state-of-the-art on two sound classification benchmarks, and perform on par with the state-of-the-art self-supervised approaches on ImageNet classification. We also demonstrate that the network is able to localize objects in both modalities, as well as perform fine-grained recognition tasks.

Key facts. Cited by 458 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 90; 2022: 96; 2023: 102; 2024: 68; 2025: 58; 2026: 44.

458 papers.

Sort:

By area:

  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  2. 2021Transformers in Vision: A Surveycited by 1,013
  3. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  4. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  5. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  6. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  7. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  8. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  9. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 264
  10. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
  11. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  12. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
38 more
  1. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  2. 2021Anticipative Video Transformercited by 200
  3. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  4. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  5. 2022Audio-Visual Segmentationcited by 151
  6. 2021Towards Long-Form Video Understandingcited by 146
  7. 2022Contrastive Audio-Visual Masked Autoencodercited by 146
  8. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 131
  9. 2022Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detectioncited by 130
  10. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 118
  11. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  12. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  13. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  14. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 87
  15. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 85
  16. 2022Localizing Visual Sounds the Easy Waycited by 82
  17. 2021Robust Audio-Visual Instance Discriminationcited by 80
  18. 2022Object discovery and representation networkscited by 80
  19. 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 80
  20. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  21. 2022HEAR: Holistic Evaluation of Audio Representationscited by 78
  22. 2023Audio-Visual Segmentation with Semanticscited by 75
  23. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 75
  24. 2023Multimodal Representation Learning by Alternating Unimodal Adaptationcited by 75
  25. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 73
  26. 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 73
  27. 2021Cross Modal Retrieval with Querybank Normalisationcited by 72
  28. 2021Positive Sample Propagation along the Audio-Visual Event Linecited by 69
  29. 2023AVSegFormer: Audio-Visual Segmentation with Transformercited by 68
  30. 2022Audio Self-supervised Learning: A Surveycited by 65
  31. 2022Learning Neural Acoustic Fieldscited by 65
  32. 2023Align and Attend: Multimodal Summarization with Dual Contrastive Lossescited by 65
  33. 2021Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICAcited by 64
  34. 2022MAViL: Masked Audio-Video Learnerscited by 59
  35. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  36. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  37. 2021Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structurecited by 53
  38. 2023On Uni-Modal Feature Learning in Supervised Multi-Modal Learningcited by 53

50 most cited of 458 All 458 (JSON)