# Multi-Source Transformer Architectures for Audiovisual Scene Classification

arXiv: https://arxiv.org/abs/2210.10212 · HTML: https://mlsanity.com/paper/2210.10212/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 14 references that are in the map.
- First posted on arXiv 18 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [Input Combination Strategies for Multi-Source Transformer Decoder](https://mlsanity.com/paper/1811.04716/) (2018)
- [A Curated Dataset of Urban Scenes for Audio-Visual Scene Analysis](https://mlsanity.com/paper/2011.00030/) (2020)
- [Audiovisual Transformer Architectures for Large-Scale Classification and Synchronization of Weakly Labeled Audio Events](https://mlsanity.com/paper/1912.02615/) (2019)
- [Audio-visual scene classification: analysis of DCASE 2021 Challenge submissions](https://mlsanity.com/paper/2105.13675/) (2021)

## Abstract (arXiv)

In this technical report, the systems we submitted for subtask 1B of the DCASE 2021 challenge, regarding audiovisual scene classification, are described in detail. They are essentially multi-source transformers employing a combination of auditory and visual features to make predictions. These models are evaluated utilizing the macro-averaged multi-class cross-entropy and accuracy metrics. In terms of the macro-averaged multi-class cross-entropy, our best model achieved a score of 0.620 on the validation data. This is slightly better than the performance of the baseline system (0.658). With regard to the accuracy measure, our best model achieved a score of 77.1% on the validation data, which is about the same as the performance obtained by the baseline system (77.0%).