# SoundNet: Learning Sound Representations from Unlabeled Video

arXiv: https://arxiv.org/abs/1610.09001 · HTML: https://mlsanity.com/paper/1610.09001/

## Key facts

- Cited by 247 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2016.

## Built on by (most cited first, 30 of 247)

- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Any-to-Any Generation via Composable Diffusion](https://mlsanity.com/paper/2305.11846/) (2023)
- [Localizing Visual Sounds the Hard Way](https://mlsanity.com/paper/2104.02691/) (2021)
- [Audio-Visual Segmentation](https://mlsanity.com/paper/2207.05042/) (2022)
- [Contrastive Audio-Visual Masked Autoencoder](https://mlsanity.com/paper/2210.07839/) (2022)
- [NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes](https://mlsanity.com/paper/2111.13260/) (2021)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos](https://mlsanity.com/paper/2104.07905/) (2021)
- [Localizing Visual Sounds the Easy Way](https://mlsanity.com/paper/2203.09324/) (2022)
- [Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval](https://mlsanity.com/paper/2112.04446/) (2021)
- [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022)
- [Audio-Visual Segmentation with Semantics](https://mlsanity.com/paper/2301.13190/) (2023)
- [A Closer Look at Weakly-Supervised Audio-Visual Source Localization](https://mlsanity.com/paper/2209.09634/) (2022)
- [LIGA-Stereo: Learning LiDAR Geometry Aware Representations for Stereo-based 3D Detector](https://mlsanity.com/paper/2108.08258/) (2021)
- [Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2104.00239/) (2021)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [Learning Neural Acoustic Fields](https://mlsanity.com/paper/2204.00628/) (2022)
- [Learning 3D Semantic Segmentation with only 2D Image Supervision](https://mlsanity.com/paper/2110.11325/) (2021)
- [MAViL: Masked Audio-Video Learners](https://mlsanity.com/paper/2212.08071/) (2022)
- [Slow-Fast Auditory Streams For Audio Recognition](https://mlsanity.com/paper/2103.03516/) (2021)
- [Mix and Localize: Localizing Sound Sources in Mixtures](https://mlsanity.com/paper/2211.15058/) (2022)
- [Distilling Audio-Visual Knowledge by Compositional Contrastive Learning](https://mlsanity.com/paper/2104.10955/) (2021)
- [Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions](https://mlsanity.com/paper/2105.04489/) (2021)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [Audio-Visual Grouping Network for Sound Localization from Mixtures](https://mlsanity.com/paper/2303.17056/) (2023)
- [ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning](https://mlsanity.com/paper/2101.10803/) (2021)
- [Contrastive Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2211.09980/) (2022)

## Abstract (arXiv)

We learn rich natural sound representations by capitalizing on large amounts of unlabeled sound data collected in the wild. We leverage the natural synchronization between vision and sound to learn an acoustic representation using two-million unlabeled videos. Unlabeled video has the advantage that it can be economically acquired at massive scales, yet contains useful signals about natural sound. We propose a student-teacher training procedure which transfers discriminative visual knowledge from well established visual recognition models into the sound modality using unlabeled video as a bridge. Our sound representation yields significant performance improvements over the state-of-the-art results on standard benchmarks for acoustic scene/object classification. Visualizations suggest some high-level semantics automatically emerge in the sound network, even though it is trained without ground truth labels.