mlsanityData to 14 Oct 2026

Multi-encoder attention-based architectures for sound recognition with partial visual assistance

arXiv 2209.12826 ↗First posted 26 Sep 2022eess.AS cs.CV cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

Large-scale sound recognition data sets typically consist of acoustic recordings obtained from multimedia libraries. As a consequence, modalities other than audio can often be exploited to improve the outputs of models designed for associated tasks. Frequently, however, not all contents are available for all samples of such a collection: For example, the original material may have been removed from the source platform at some point, and therefore, non-auditory features can no longer be acquired. We demonstrate that a multi-encoder framework can be employed to deal with this issue by applying this method to attention-based deep learning systems, which are currently part of the state of the art in the domain of sound recognition. More specifically, we show that the proposed model extension can successfully be utilized to incorporate partially available visual information into the operational procedures of such networks, which normally only use auditory features during training and inference. Experimentally, we verify that the considered approach leads to improved predictions in a number of evaluation scenarios pertaining to audio tagging and sound event detection. Additionally, we scrutinize some properties and limitations of the presented technique.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 53 references that are in the map. First posted on arXiv 26 Sep 2022.

Builds on

23 of 53 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  9. 2016Layer Normalizationcited by 5,669
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
11 more
  1. 2020On Layer Normalization in the Transformer Architecturecited by 922
  2. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  3. 2016Language Modeling with Gated Convolutional Networkscited by 702
  4. 2017Convolutional Sequence to Sequence Learningcited by 552
  5. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
  6. 2019A Framework for the Robust Evaluation of Sound Event Detectioncited by 29
  7. 2020Temporally Coherent Embeddings for Self-Supervised Video Representation Learningcited by 17
  8. 2019Audiovisual Transformer Architectures for Large-Scale Classification and Synchronization of Weakly Labeled Audio Eventscited by 9
  9. 2020Training Sound Event Detection On A Heterogeneous Datasetcited by 7
  10. 2021Multi-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognitioncited by 6
  11. 2021Audiovisual transfer learning for audio tagging and sound event detectioncited by 6

Built on by

Not cited yet.