# Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data

arXiv: https://arxiv.org/abs/2006.01595 · HTML: https://mlsanity.com/paper/2006.01595/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2020.

## Built on by (most cited first, 27 of 27)

- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [Contrastive Audio-Visual Masked Autoencoder](https://mlsanity.com/paper/2210.07839/) (2022)
- [Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2104.00239/) (2021)
- [MAViL: Masked Audio-Video Learners](https://mlsanity.com/paper/2212.08071/) (2022)
- [PolyViT: Co-training Vision Transformers on Images, Videos and Audio](https://mlsanity.com/paper/2111.12993/) (2021)
- [On Uni-Modal Feature Learning in Supervised Multi-Modal Learning](https://mlsanity.com/paper/2305.01233/) (2023)
- [Contrastive Positive Sample Propagation along the Audio-Visual Event Line](https://mlsanity.com/paper/2211.09980/) (2022)
- [Audiovisual Masked Autoencoders](https://mlsanity.com/paper/2212.05922/) (2022)
- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation](https://mlsanity.com/paper/2409.19132/) (2024)
- [Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Models](https://mlsanity.com/paper/2207.07646/) (2022)
- [UAVM: Towards Unifying Audio and Visual Models](https://mlsanity.com/paper/2208.00061/) (2022)
- [Temporal and cross-modal attention for audio-visual zero-shot learning](https://mlsanity.com/paper/2207.09966/) (2022)
- [Siamese Vision Transformers are Scalable Audio-visual Learners](https://mlsanity.com/paper/2403.19638/) (2024)
- [Zorro: the masked multimodal transformer](https://mlsanity.com/paper/2301.09595/) (2023)
- [Exploring Fine-Grained Audiovisual Categorization with the SSW60 Dataset](https://mlsanity.com/paper/2207.10664/) (2022)
- [EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning](https://mlsanity.com/paper/2403.09502/) (2024)
- [Space-Time Memory Network for Sounding Object Localization in Videos](https://mlsanity.com/paper/2111.05526/) (2021)
- [Audio-visual Generalized Zero-shot Learning the Easy Way](https://mlsanity.com/paper/2407.13095/) (2024)
- [A dataset for Audio-Visual Sound Event Detection in Movies](https://mlsanity.com/paper/2302.07315/) (2023)
- [Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning](https://mlsanity.com/paper/2304.05600/) (2023)
- [Where and When: Space-Time Attention for Audio-Visual Explanations](https://mlsanity.com/paper/2105.01517/) (2021)
- [Text-to-feature diffusion for audio-visual few-shot learning](https://mlsanity.com/paper/2309.03869/) (2023)
- [Dynamic Inter-Class Confusion-Aware Encoder for Audio-Visual Fusion in Human Activity Recognition](https://mlsanity.com/paper/2507.09323/) (2025)
- [Social-MAE: A Transformer-Based Multimodal Autoencoder for Face and Voice](https://mlsanity.com/paper/2508.17502/) (2025)

## Abstract (arXiv)

Recognizing sounds is a key aspect of computational audio scene analysis and machine perception. In this paper, we advocate that sound recognition is inherently a multi-modal audiovisual task in that it is easier to differentiate sounds using both the audio and visual modalities as opposed to one or the other. We present an audiovisual fusion model that learns to recognize sounds from weakly labeled video recordings. The proposed fusion model utilizes an attention mechanism to dynamically combine the outputs of the individual audio and visual models. Experiments on the large scale sound events dataset, AudioSet, demonstrate the efficacy of the proposed model, which outperforms the single-modal models, and state-of-the-art fusion and multi-modal models. We achieve a mean Average Precision (mAP) of 46.16 on Audioset, outperforming prior state of the art by approximately +4.35 mAP (relative: 10.4%).