mlsanityData to 7 Oct 2026

Ambient Sound Provides Supervision for Visual Learning

arXiv 1608.07017 ↗First posted 25 Aug 2016cs.CVcited by 77

Not on the map. Why?

Abstract (arXiv)

The sound of crashing waves, the roar of fast-moving cars -- sound conveys important information about the objects in our surroundings. In this work, we show that ambient sounds can be used as a supervisory signal for learning visual models. To demonstrate this, we train a convolutional neural network to predict a statistical summary of the sound associated with a video frame. We show that, through this process, the network learns a representation that conveys information about objects and scenes. We evaluate this representation on several recognition tasks, finding that its performance is comparable to that of other state-of-the-art unsupervised learning methods. Finally, we show through visualizations that the network learns units that are selective to objects that are often associated with characteristic sounds.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Aug 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 14; 2023: 15; 2024: 10; 2025: 8; 2026: 4.

77 papers.

Sort:

By area:

  1. 2023A Cookbook of Self-Supervised Learningcited by 248
  2. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  3. 2021NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenescited by 119
  4. 2022Localizing Visual Sounds the Easy Waycited by 81
  5. 2021Cross Modal Retrieval with Querybank Normalisationcited by 71
  6. 2021Learning 3D Semantic Segmentation with only 2D Image Supervisioncited by 60
  7. 2021Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptionscited by 53
  8. 2023Audio-Visual Grouping Network for Sound Localization from Mixturescited by 52
  9. 2021EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillationcited by 46
  10. 2021Time-Equivariant Contrastive Video Representation Learningcited by 44
  11. 2023Sound to Visual Scene Generation by Audio-to-Visual Latent Alignmentcited by 41
  12. 2021Improving Multi-Modal Learning with Uni-Modal Teacherscited by 40
38 more
  1. 2023Egocentric Audio-Visual Object Localizationcited by 40
  2. 2022Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenescited by 31
  3. 2023Sound Source Localization is All about Cross-Modal Alignmentcited by 30
  4. 2024Text-to-Audio Generation Synchronized with Videoscited by 28
  5. 2022Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Datacited by 27
  6. 2021Multimodal Knowledge Expansioncited by 26
  7. 2021Robust Object Detection via Instance-Level Temporal Cycle Confusioncited by 24
  8. 2021Self-supervised Audiovisual Representation Learning for Remote Sensing Datacited by 24
  9. 2022Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Workscited by 24
  10. 2023Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modelingcited by 24
  11. 2021Can audio-visual integration strengthen robustness under multimodal attacks?cited by 23
  12. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  13. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  14. 2021Hear Me Out: Fusional Approaches for Audio Augmented Temporal Action Localizationcited by 21
  15. 2023Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Bankscited by 20
  16. 2021Contrastive Learning of Image Representations with Cross-Video Cycle-Consistencycited by 19
  17. 2023Hyperbolic Audio-visual Zero-shot Learningcited by 19
  18. 2023Audio-Visual Contrastive Learning with Temporal Self-Supervisioncited by 18
  19. 2023Egocentric Auditory Attention Localization in Conversationscited by 18
  20. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16
  21. 2023Learning Tri-modal Embeddings for Zero-Shot Soundscape Mappingcited by 14
  22. 2025CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignmentcited by 14
  23. 2021Composable Augmentation Encoding for Video Representation Learningcited by 13
  24. 2021Audio-visual Representation Learning for Anomaly Events Detection in Crowdscited by 13
  25. 2024Siamese Vision Transformers are Scalable Audio-visual Learnerscited by 12
  26. 2025Dynamic Multi-Target Fusion for Efficient Audio-Visual Navigationcited by 12
  27. 2024SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Soundcited by 10
  28. 2024CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localizationcited by 10
  29. 2021Temporal Knowledge Consistency for Unsupervised Visual Representation Learningcited by 8
  30. 2021Dynamic Modeling of Hand-Object Interactions via Tactile Sensingcited by 8
  31. 2025VGGSounder: Audio-Visual Evaluations for Foundation Modelscited by 8
  32. 2022Leveraging the Video-level Semantic Consistency of Event for Audio-visual Event Localizationcited by 7
  33. 2023Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and Audiocited by 6
  34. 2021Cross-Modal Attention Consistency for Video-Audio Unsupervised Learningcited by 5
  35. 2024The Sound of Water: Inferring Physical Properties from Pouring Liquidscited by 5
  36. 2024Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignmentcited by 5
  37. 2021Weakly-supervised Audio-visual Sound Source Detection and Separationcited by 4
  38. 2021Binaural SoundNet: Predicting Semantics, Depth and Motion with Binaural Soundscited by 4

50 most cited of 77 All 77 (JSON)