mlsanityData to 14 Oct 2026

SoundNet: Learning Sound Representations from Unlabeled Video

arXiv 1610.09001 ↗First posted 27 Oct 2016cs.CV cs.LG cs.SDcited by 247

Not on the map. Why?

Abstract (arXiv)

We learn rich natural sound representations by capitalizing on large amounts of unlabeled sound data collected in the wild. We leverage the natural synchronization between vision and sound to learn an acoustic representation using two-million unlabeled videos. Unlabeled video has the advantage that it can be economically acquired at massive scales, yet contains useful signals about natural sound. We propose a student-teacher training procedure which transfers discriminative visual knowledge from well established visual recognition models into the sound modality using unlabeled video as a bridge. Our sound representation yields significant performance improvements over the state-of-the-art results on standard benchmarks for acoustic scene/object classification. Visualizations suggest some high-level semantics automatically emerge in the sound network, even though it is trained without ground truth labels.

Key facts. Cited by 247 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Oct 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 62; 2022: 61; 2023: 56; 2024: 31; 2025: 20; 2026: 17.

247 papers.

Sort:

By area:

  1. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  2. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 264
  3. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
  4. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  5. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  6. 2021Localizing Visual Sounds the Hard Waycited by 170
  7. 2022Audio-Visual Segmentationcited by 151
  8. 2022Contrastive Audio-Visual Masked Autoencodercited by 146
  9. 2021NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenescited by 119
  10. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  11. 2021Ego-Exo: Transferring Visual Representations from Third-person to First-person Videoscited by 99
  12. 2022Localizing Visual Sounds the Easy Waycited by 82
38 more
  1. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  2. 2022HEAR: Holistic Evaluation of Audio Representationscited by 78
  3. 2023Audio-Visual Segmentation with Semanticscited by 75
  4. 2022A Closer Look at Weakly-Supervised Audio-Visual Source Localizationcited by 73
  5. 2021LIGA-Stereo: Learning LiDAR Geometry Aware Representations for Stereo-based 3D Detectorcited by 70
  6. 2021Positive Sample Propagation along the Audio-Visual Event Linecited by 69
  7. 2022Audio Self-supervised Learning: A Surveycited by 65
  8. 2022Learning Neural Acoustic Fieldscited by 65
  9. 2021Learning 3D Semantic Segmentation with only 2D Image Supervisioncited by 60
  10. 2022MAViL: Masked Audio-Video Learnerscited by 59
  11. 2021Slow-Fast Auditory Streams For Audio Recognitioncited by 56
  12. 2022Mix and Localize: Localizing Sound Sources in Mixturescited by 54
  13. 2021Distilling Audio-Visual Knowledge by Compositional Contrastive Learningcited by 53
  14. 2021Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptionscited by 53
  15. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  16. 2023Audio-Visual Grouping Network for Sound Localization from Mixturescited by 52
  17. 2021ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learningcited by 50
  18. 2022Contrastive Positive Sample Propagation along the Audio-Visual Event Linecited by 50
  19. 2023Learning Audio-Visual Source Localization via False Negative Aware Contrastive Learningcited by 47
  20. 2024OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flowscited by 46
  21. 2021There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledgecited by 44
  22. 2022The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillationcited by 44
  23. 2022Discrete Contrastive Diffusion for Cross-Modal Music and Image Generationcited by 44
  24. 2022Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisationcited by 44
  25. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  26. 2021Improving Multi-Modal Learning with Uni-Modal Teacherscited by 41
  27. 2022Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipationcited by 41
  28. 2023Sound to Visual Scene Generation by Audio-to-Visual Latent Alignmentcited by 41
  29. 2023Egocentric Audio-Visual Object Localizationcited by 40
  30. 2022ECLIPSE: Efficient Long-range Video Retrieval using Sight and Soundcited by 39
  31. 2022Temporal Sentence Grounding in Videos: A Survey and Future Directionscited by 37
  32. 2022Quantized GAN for Complex Music Generation from Dance Videoscited by 37
  33. 2021ContIG: Self-supervised Multimodal Contrastive Learning for Medical Imaging with Geneticscited by 36
  34. 2021VPN++: Rethinking Video-Pose embeddings for understanding Activities of Daily Livingcited by 35
  35. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  36. 2022Probabilistic Representations for Video Contrastive Learningcited by 34
  37. 2021Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretrainingcited by 33
  38. 2021Class-aware Sounding Objects Localization via Audiovisual Correspondencecited by 32

50 most cited of 247 All 247 (JSON)