mlsanityData to 7 Oct 2026

A study on joint modeling and data augmentation of multi-modalities for audio-visual scene classification

arXiv 2203.04114 ↗First posted 7 Mar 2022cs.MM cs.CV cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose two techniques, namely joint modeling and data augmentation, to improve system performances for audio-visual scene classification (AVSC). We employ pre-trained networks trained only on image data sets to extract video embedding; whereas for audio embedding models, we decide to train them from scratch. We explore different neural network architectures for joint modeling to effectively combine the video and audio modalities. Moreover, data augmentation strategies are investigated to increase audio-visual training set size. For the video modality the effectiveness of several operations in RandAugment is verified. An audio-video joint mixup scheme is proposed to further improve AVSC performances. Evaluated on the development set of TAU Urban Audio Visual Scenes 2021, our final system can achieve the best accuracy of 94.2% among all single AVSC systems submitted to DCASE 2021 Task 1b.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 37 references that are in the map. First posted on arXiv 7 Mar 2022.

Builds on

18 of 37 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2017Squeeze-and-Excitation Networkscited by 4,763
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  9. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  10. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  11. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  12. 2020ResNeSt: Split-Attention Networkscited by 445
6 more
  1. 2018A multi-device dataset for urban acoustic scene classificationcited by 75
  2. 2020Acoustic scene classification in DCASE 2020 Challenge: generalization across devices and low complexity solutionscited by 48
  3. 2020A Curated Dataset of Urban Scenes for Audio-Visual Scene Analysiscited by 16
  4. 2019Multiple instance dense connected convolution neural network for aerial image scene classificationcited by 15
  5. 2020A Two-Stage Approach to Device-Robust Acoustic Scene Classificationcited by 9
  6. 2015A Spatial Layout and Scale Invariant Feature Representation for Indoor Scene Classificationcited by 5

Built on by

Not cited yet.