mlsanityData to 7 Oct 2026

Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data

arXiv 2006.01595 ↗First posted 29 May 2020eess.AS cs.LG cs.SDcited by 27

Not on the map. Why?

Abstract (arXiv)

Recognizing sounds is a key aspect of computational audio scene analysis and machine perception. In this paper, we advocate that sound recognition is inherently a multi-modal audiovisual task in that it is easier to differentiate sounds using both the audio and visual modalities as opposed to one or the other. We present an audiovisual fusion model that learns to recognize sounds from weakly labeled video recordings. The proposed fusion model utilizes an attention mechanism to dynamically combine the outputs of the individual audio and visual models. Experiments on the large scale sound events dataset, AudioSet, demonstrate the efficacy of the proposed model, which outperforms the single-modal models, and state-of-the-art fusion and multi-modal models. We achieve a mean Average Precision (mAP) of 46.16 on Audioset, outperforming prior state of the art by approximately +4.35 mAP (relative: 10.4%).

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 8; 2022: 8; 2023: 5; 2024: 4; 2025: 2.

27 papers.

Sort:

  1. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  2. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  3. 2021What Makes Multi-modal Learning Better than Single (Provably)cited by 151
  4. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  5. 2021Positive Sample Propagation along the Audio-Visual Event Linecited by 69
  6. 2021PolyViT: Co-training Vision Transformers on Images, Videos and Audiocited by 58
  7. 2022MAViL: Masked Audio-Video Learnerscited by 56
  8. 2023On Uni-Modal Feature Learning in Supervised Multi-Modal Learningcited by 53
  9. 2022Contrastive Positive Sample Propagation along the Audio-Visual Event Linecited by 50
  10. 2022Audiovisual Masked Autoencoderscited by 47
  11. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  12. 2024From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generationcited by 23
15 more
  1. 2022Multimodal Open-Vocabulary Video Classification via Pre-Trained Vision and Language Modelscited by 22
  2. 2022UAVM: Towards Unifying Audio and Visual Modelscited by 20
  3. 2022Temporal and cross-modal attention for audio-visual zero-shot learningcited by 19
  4. 2023Zorro: the masked multimodal transformercited by 13
  5. 2022Exploring Fine-Grained Audiovisual Categorization with the SSW60 Datasetcited by 12
  6. 2024Siamese Vision Transformers are Scalable Audio-visual Learnerscited by 12
  7. 2021Space-Time Memory Network for Sounding Object Localization in Videoscited by 10
  8. 2024EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learningcited by 10
  9. 2024Audio-visual Generalized Zero-shot Learning the Easy Waycited by 7
  10. 2023A dataset for Audio-Visual Sound Event Detection in Moviescited by 5
  11. 2023Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learningcited by 3
  12. 2021Where and When: Space-Time Attention for Audio-Visual Explanationscited by 1
  13. 2023Text-to-feature diffusion for audio-visual few-shot learningcited by 1
  14. 2025Dynamic Inter-Class Confusion-Aware Encoder for Audio-Visual Fusion in Human Activity Recognitioncited by 0
  15. 2025Social-MAE: A Transformer-Based Multimodal Autoencoder for Face and Voicecited by 0